From 95015de73396fec3b24ac0da526c11ecf9a3165e Mon Sep 17 00:00:00 2001 From: Mateo Wang <277851410+mateo-berri@users.noreply.github.com> Date: Thu, 28 May 2026 09:14:57 -0700 Subject: [PATCH] feat: add support for claude code goal mode for bedrock opus output config (#28898) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * feat: support goal mode for claude on bedrock * fix failing lint test * addressing greptile comments * fixing failed test * address greptile: copy output_config and warn on dropped converse format * fix(bedrock): skip redundant output_config normalization on Converse reasoning_effort path When reasoning_effort is mapped via _handle_reasoning_effort_parameter, the resulting output_config is already normalized via normalize_bedrock_opus_output_config_effort. Mark it as normalized so _prepare_request_params can skip the redundant call (and the associated get_model_info lookup) on every request. Co-authored-by: Yassin Kortam * test(reasoning-effort-grid): reflect Bedrock opus-4-6 xhigh→max clamping * fix(bedrock): stop leaking output_config marker and message-content mutation * fix(bedrock): guard effort key access in normalize_bedrock_opus_output_config_effort Defensively check that 'effort' is a valid key in _BEDROCK_OUTPUT_CONFIG_EFFORT_ORDER before indexing, to prevent a KeyError if the hardcoded guard tuple ever drifts from the order dict's keys. Co-authored-by: Yassin Kortam * fix(bedrock): drop dead second clause in effort normalization guard The 'effort not in _BEDROCK_OUTPUT_CONFIG_EFFORT_ORDER' check is unreachable once 'effort not in ("xhigh", "max")' has been ruled out, since both literals are present in the order dict. Keep the literal membership check and let the dict lookups below speak for themselves. * fix(bedrock): clamp output_config.effort against ceiling for any known value The early return when effort was not 'xhigh'/'max' meant a ceiling of 'low' or 'medium' would silently forward an out-of-range value. Gate on the known effort ordering instead so the ceiling comparison runs for every recognized effort. * test(grid_spec): use _CAPS_OPUS_4_7 for non-Bedrock opus-4-6 entries claude-opus-4-6 now declares supports_xhigh_reasoning_effort in the model map, so production accepts xhigh on Azure AI and Vertex AI routes. Update those grid_spec entries to match production capabilities so expected() predicts 200 for xhigh instead of 400. Co-authored-by: Yassin Kortam * test(grid_spec): revert xhigh caps for non-Bedrock opus-4-6 azure_ai/claude-opus-4-6 and vertex_ai/claude-opus-4-6 do not declare supports_xhigh_reasoning_effort in model_prices_and_context_window.json. Azure AI upstream rejects xhigh with HTTP 400 ("Supported levels: high, low, max, medium"). Restore _CAPS_4_6 so the grid predicts 400 for xhigh, matching production capabilities. * fix: stop advertising xhigh effort on Opus 4.5/4.6 Only Opus 4.7 supports the xhigh reasoning effort level. Remove the supports_xhigh_reasoning_effort flag from every Opus 4.5 and Opus 4.6 entry (direct Anthropic, Bedrock, and regional variants) in both model catalog files. On the direct Anthropic path there is no effort clamp, so flagging 4.5/4.6 as xhigh-capable caused litellm to forward xhigh to a model that rejects it (and made get_model_info misreport the capability). xhigh now correctly degrades to high / raises on those models. Bedrock graceful degradation for Claude Code goal mode is unaffected: it relies solely on the bedrock_output_config_effort_ceiling clamp (4.5->high, 4.6->max, 4.7->xhigh), which runs before validation, so xhigh requests to older Bedrock Opus models are still silently lowered rather than rejected. Update effort-gating tests to reflect that 4.5/4.6 no longer accept xhigh. * fix: clamp xhigh effort on Bedrock Invoke /v1/messages instead of rejecting Claude Code "goal mode" sends output_config.effort=xhigh over the Anthropic /v1/messages API, which routes Bedrock models through AmazonAnthropicClaudeMessagesConfig. That path validated effort against the model's native capability and raised 400 for xhigh on Opus 4.6, while the chat-completions paths (Converse + Invoke) already clamp xhigh to the model's bedrock_output_config_effort_ceiling. That asymmetry broke goal mode on the exact API surface Claude Code uses. Apply the same ceiling clamp on the messages path before the shared effort gate runs, so xhigh degrades to max on Opus 4.6 (and stays xhigh on 4.7). Scoped to adaptive-thinking models and to models that declare a ceiling, so Sonnet 4.6 (no ceiling) and Opus 4.5 (budget mode) are unaffected and still reject xhigh. * fix(bedrock): preserve user output_config when applying reasoning_effort - Converse path: merge mapped effort into existing output_config via setdefault instead of overwriting it, matching the Anthropic Messages path. Prevents user-supplied output_config.format from being silently dropped when reasoning_effort is also provided. - tests: clear _get_local_model_cost_map lru_cache in the autouse fixture alongside get_bedrock_response_stream_shape to avoid stale cache leakage between tests. Co-authored-by: Yassin Kortam * fix(bedrock): pre-clamp reasoning_effort for chat invoke; correct test caps - Add _clamp_adaptive_reasoning_effort_for_bedrock to AmazonAnthropicClaudeConfig so raw reasoning_effort=xhigh degrades to the model's bedrock effort ceiling before AnthropicConfig.map_openai_params converts it to output_config. Mirrors converse path (_handle_reasoning_effort_parameter) and messages path (_clamp_adaptive_reasoning_effort_for_bedrock) so the three Bedrock paths are consistent. - grid_spec: restore caps=_CAPS_4_6 for Bedrock converse/invoke Opus 4.6 entries so the test reflects the model's actual JSON capabilities. Teach expected() to bypass the xhigh/max cap check when bedrock_effort_ceiling will clamp the wire effort, so the test still passes for Bedrock's graceful degradation contract without lying about native model caps. Co-authored-by: Yassin Kortam --------- Co-authored-by: Dennis Henry Co-authored-by: Cursor Agent Co-authored-by: Yassin Kortam --- litellm/llms/anthropic/chat/transformation.py | 5 +- .../messages/transformation.py | 9 +- .../bedrock/chat/converse_transformation.py | 60 +++- .../anthropic_claude3_transformation.py | 57 +++- litellm/llms/bedrock/common_utils.py | 138 ++++++++ .../anthropic_claude3_transformation.py | 308 ++++++++++-------- ...odel_prices_and_context_window_backup.json | 57 +++- litellm/types/llms/anthropic.py | 3 +- litellm/types/utils.py | 3 + litellm/utils.py | 3 + model_prices_and_context_window.json | 57 +++- .../reasoning_effort_grid/grid_spec.py | 35 +- .../test_anthropic_chat_transformation.py | 59 +++- ...t_anthropic_messages_structured_outputs.py | 33 ++ .../test_reasoning_effort_translation.py | 54 ++- ...ations_anthropic_claude3_transformation.py | 55 ++++ .../chat/test_converse_transformation.py | 127 ++++++++ .../test_anthropic_claude3_transformation.py | 169 +++++++++- .../llms/bedrock/test_bedrock_common_utils.py | 47 ++- tests/test_litellm/test_utils.py | 6 +- 20 files changed, 1068 insertions(+), 217 deletions(-) diff --git a/litellm/llms/anthropic/chat/transformation.py b/litellm/llms/anthropic/chat/transformation.py index 0b56eb86d9..1e5118dc41 100644 --- a/litellm/llms/anthropic/chat/transformation.py +++ b/litellm/llms/anthropic/chat/transformation.py @@ -1793,7 +1793,10 @@ class AnthropicConfig(AnthropicModelInfo, BaseConfig): self._ensure_context_management_beta_header( headers, optional_params["context_management"] ) - if optional_params.get("output_format") is not None: + output_config = optional_params.get("output_config") + if optional_params.get("output_format") is not None or ( + isinstance(output_config, dict) and output_config.get("format") is not None + ): self._ensure_beta_header( headers, ANTHROPIC_BETA_HEADER_VALUES.STRUCTURED_OUTPUT_2025_09_25.value ) diff --git a/litellm/llms/anthropic/experimental_pass_through/messages/transformation.py b/litellm/llms/anthropic/experimental_pass_through/messages/transformation.py index 15f404d3f5..f94232fa45 100644 --- a/litellm/llms/anthropic/experimental_pass_through/messages/transformation.py +++ b/litellm/llms/anthropic/experimental_pass_through/messages/transformation.py @@ -427,8 +427,13 @@ class AnthropicMessagesConfig(BaseAnthropicMessagesConfig): ANTHROPIC_BETA_HEADER_VALUES.CONTEXT_MANAGEMENT_2025_06_27.value ) - # Check for structured outputs - if optional_params.get("output_format") is not None: + # Check for structured outputs. Anthropic's newer request shape nests + # the schema under output_config.format; the older top-level + # output_format remains supported for backwards compatibility. + output_config = optional_params.get("output_config") + if optional_params.get("output_format") is not None or ( + isinstance(output_config, dict) and output_config.get("format") is not None + ): beta_values.add( ANTHROPIC_BETA_HEADER_VALUES.STRUCTURED_OUTPUT_2025_09_25.value ) diff --git a/litellm/llms/bedrock/chat/converse_transformation.py b/litellm/llms/bedrock/chat/converse_transformation.py index ae100eda8d..d58d2e2759 100644 --- a/litellm/llms/bedrock/chat/converse_transformation.py +++ b/litellm/llms/bedrock/chat/converse_transformation.py @@ -78,6 +78,7 @@ from ..common_utils import ( get_anthropic_beta_from_headers, get_bedrock_tool_name, is_claude_4_5_on_bedrock, + normalize_bedrock_opus_output_config_effort, ) # Computer use tool prefixes supported by Bedrock @@ -448,10 +449,20 @@ class AmazonConverseConfig(BaseConfig): value=reasoning_effort, llm_provider="bedrock_converse", ) + existing_output_config = optional_params.get("output_config") + if not isinstance(existing_output_config, dict): + existing_output_config = {} + existing_output_config.setdefault("effort", mapped_effort) + normalize_bedrock_opus_output_config_effort( + model=model, + output_config=existing_output_config, + ) + mapped_effort = existing_output_config["effort"] self._validate_anthropic_adaptive_effort( model=model, effort=mapped_effort ) - optional_params["output_config"] = {"effort": mapped_effort} + optional_params["output_config"] = existing_output_config + optional_params["_output_config_normalized"] = True @staticmethod def _validate_anthropic_adaptive_effort(model: str, effort: str) -> None: @@ -1201,6 +1212,12 @@ class AmazonConverseConfig(BaseConfig): self, optional_params: dict, model: str ) -> Tuple[dict, dict, dict, Optional[OutputConfigBlock]]: """Prepare and separate request parameters.""" + # Consume the internal ``_output_config_normalized`` marker set by + # ``_handle_reasoning_effort_parameter`` so it does not linger on the + # caller's ``optional_params`` after the transformation returns. + anthropic_output_config_already_normalized = bool( + optional_params.pop("_output_config_normalized", False) + ) # Filter out exception objects before deepcopy to prevent deepcopy failures # Exceptions should not be stored in optional_params (this is a defensive fix) cleaned_params = filter_exceptions_from_params(optional_params) @@ -1219,8 +1236,17 @@ class AmazonConverseConfig(BaseConfig): # Anthropic-only ``output_config`` (snake_case) — re-attached to # ``additionalModelRequestFields`` for Anthropic models below. The - # Bedrock-native ``outputConfig`` (camelCase) is handled separately. + # structured-output ``format`` subfield is consumed into Bedrock's + # native ``outputConfig`` (camelCase), which is handled separately. anthropic_output_config = inference_params.pop("output_config", None) + output_config_format = None + if isinstance(anthropic_output_config, dict): + anthropic_output_config = dict(anthropic_output_config) + candidate_output_config_format = anthropic_output_config.pop("format", None) + if isinstance(candidate_output_config_format, dict): + output_config_format = candidate_output_config_format + if not anthropic_output_config: + anthropic_output_config = None # Extract requestMetadata before processing other parameters request_metadata = inference_params.pop("requestMetadata", None) @@ -1230,6 +1256,30 @@ class AmazonConverseConfig(BaseConfig): output_config: Optional[OutputConfigBlock] = inference_params.pop( "outputConfig", None ) + base_model = BedrockModelInfo.get_base_model(model) + if ( + output_config is None + and output_config_format is not None + and output_config_format.get("type") == "json_schema" + and base_model.startswith("anthropic") + and self._supports_native_structured_outputs( + model, self.custom_llm_provider + ) + ): + output_config = self._create_output_config_for_response_format( + json_schema=output_config_format.get("schema"), + name=output_config_format.get("name"), + description=output_config_format.get("description"), + ) + elif output_config is None and output_config_format is not None: + litellm.verbose_logger.warning( + "Bedrock Converse: dropping `output_config.format` for model=%s — " + "model does not advertise `supports_native_structured_output` in " + "model_prices_and_context_window.json. The schema will not be " + "enforced; pass `response_format` to use the synthetic tool-call " + "fallback.", + model, + ) # keep supported params in 'inference_params', and set all model-specific params in 'additional_request_params' additional_request_params = { @@ -1275,7 +1325,6 @@ class AmazonConverseConfig(BaseConfig): if anthropic_output_config is not None and isinstance( anthropic_output_config, dict ): - base_model = BedrockModelInfo.get_base_model(model) if base_model.startswith("anthropic"): if ( litellm.drop_params is True @@ -1286,6 +1335,11 @@ class AmazonConverseConfig(BaseConfig): model, ) else: + if not anthropic_output_config_already_normalized: + normalize_bedrock_opus_output_config_effort( + model=model, + output_config=anthropic_output_config, + ) effort = anthropic_output_config.get("effort") if effort is not None: self._validate_anthropic_adaptive_effort( diff --git a/litellm/llms/bedrock/chat/invoke_transformations/anthropic_claude3_transformation.py b/litellm/llms/bedrock/chat/invoke_transformations/anthropic_claude3_transformation.py index d9599b8b9c..a13336b6c8 100644 --- a/litellm/llms/bedrock/chat/invoke_transformations/anthropic_claude3_transformation.py +++ b/litellm/llms/bedrock/chat/invoke_transformations/anthropic_claude3_transformation.py @@ -16,8 +16,11 @@ from litellm.llms.bedrock.chat.invoke_transformations.base_invoke_transformation AmazonInvokeConfig, ) from litellm.llms.bedrock.common_utils import ( + convert_bedrock_invoke_output_format_to_inline_schema, get_anthropic_beta_from_headers, + normalize_bedrock_opus_output_config_effort, normalize_tool_input_schema_types_for_bedrock_invoke, + pop_bedrock_invoke_output_config_format, remove_custom_field_from_tools, ) from litellm.types.llms.anthropic import ANTHROPIC_TOOL_SEARCH_BETA_HEADER @@ -75,6 +78,17 @@ class AmazonAnthropicClaudeConfig(AmazonInvokeConfig, AnthropicConfig): # Use a model name that forces tool-based approach model = "claude-3-sonnet-20240229" + # Clamp ``reasoning_effort`` to the Bedrock effort ceiling before the + # parent mapping converts it to ``output_config.effort`` and the + # downstream effort gate runs. Mirrors the converse path's + # ``_handle_reasoning_effort_parameter`` and the messages path's + # ``_clamp_adaptive_reasoning_effort_for_bedrock`` so adaptive Claude + # requests degrade ``xhigh`` -> ``max`` rather than 400-ing on + # models like Opus 4.6 that don't natively advertise xhigh. + self._clamp_adaptive_reasoning_effort_for_bedrock( + model=original_model, params=non_default_params + ) + optional_params = AnthropicConfig.map_openai_params( self, non_default_params, @@ -88,6 +102,27 @@ class AmazonAnthropicClaudeConfig(AmazonInvokeConfig, AnthropicConfig): return optional_params + @staticmethod + def _clamp_adaptive_reasoning_effort_for_bedrock(model: str, params: dict) -> None: + """Lower ``reasoning_effort`` to the Bedrock effort ceiling before mapping. + + Bedrock's adaptive Claude models accept the OpenAI-style + ``reasoning_effort`` tier, but the request validator can reject tiers + the model does not natively advertise (e.g. ``xhigh`` on Opus 4.6). + Clamp the raw tier to the model's + ``bedrock_output_config_effort_ceiling`` so Claude Code "goal mode" + keeps working. Non-adaptive models and models without a ceiling are + left untouched. + """ + if not AnthropicConfig._is_adaptive_thinking_model(model): + return + effort = params.get("reasoning_effort") + if not isinstance(effort, str): + return + clamped = {"effort": effort} + normalize_bedrock_opus_output_config_effort(model=model, output_config=clamped) + params["reasoning_effort"] = clamped["effort"] + def transform_request( self, model: str, @@ -157,6 +192,13 @@ class AmazonAnthropicClaudeConfig(AmazonInvokeConfig, AnthropicConfig): for k, v in optional_params.items() if k not in self.aws_authentication_params } + output_config = filtered_params.get("output_config") + if isinstance(output_config, dict): + filtered_params["output_config"] = dict(output_config) + normalize_bedrock_opus_output_config_effort( + model=model, + output_config=filtered_params["output_config"], + ) filtered_params = self._normalize_bedrock_tool_search_tools(filtered_params) anthropic_request = AnthropicConfig.transform_request( @@ -170,7 +212,20 @@ class AmazonAnthropicClaudeConfig(AmazonInvokeConfig, AnthropicConfig): anthropic_request.pop("model", None) anthropic_request.pop("stream", None) - anthropic_request.pop("output_format", None) + output_format = anthropic_request.pop("output_format", None) + output_config_format = pop_bedrock_invoke_output_config_format( + anthropic_request + ) + if output_format: + convert_bedrock_invoke_output_format_to_inline_schema( + output_format=output_format, + request_body=anthropic_request, + ) + elif output_config_format: + convert_bedrock_invoke_output_format_to_inline_schema( + output_format=output_config_format, + request_body=anthropic_request, + ) if not ( _supports_factory( model=model, diff --git a/litellm/llms/bedrock/common_utils.py b/litellm/llms/bedrock/common_utils.py index 4f4729e401..bdc5da321c 100644 --- a/litellm/llms/bedrock/common_utils.py +++ b/litellm/llms/bedrock/common_utils.py @@ -34,6 +34,15 @@ class BedrockError(BaseLLMException): # Lazy import cache to avoid circular imports and performance impact _get_model_info = None +BedrockOutputConfigEffort = Literal["low", "medium", "high", "max", "xhigh"] +_BEDROCK_OUTPUT_CONFIG_EFFORT_ORDER: Dict[BedrockOutputConfigEffort, int] = { + "low": 0, + "medium": 1, + "high": 2, + "max": 3, + "xhigh": 4, +} + def get_cached_model_info(): """ @@ -51,6 +60,79 @@ def get_cached_model_info(): return _get_model_info +@functools.lru_cache(maxsize=1) +def _get_local_model_cost_map() -> Dict: + from litellm.litellm_core_utils.get_model_cost_map import GetModelCostMap + + return GetModelCostMap.load_local_model_cost_map() + + +def pop_bedrock_invoke_output_config_format(request_body: Dict) -> Optional[Dict]: + """ + Remove and return Anthropic's nested ``output_config.format`` field. + + Bedrock Invoke paths convert the schema to inline message text. Any remaining + ``output_config`` keys, such as ``effort``, are left in place. + """ + output_config = request_body.get("output_config") + if not isinstance(output_config, dict): + return None + + output_format = output_config.pop("format", None) + if not output_config: + request_body.pop("output_config", None) + + if isinstance(output_format, dict): + return output_format + return None + + +def convert_bedrock_invoke_output_format_to_inline_schema( + output_format: Dict, + request_body: Dict, +) -> None: + """ + Embed an Anthropic structured-output schema into the last user message. + + Bedrock Invoke does not support ``output_format`` directly, so the schema is + appended to the final user message for prompt-engineered structured output. + The caller's ``messages`` list, message dict, and content list are not + mutated; a fresh ``messages`` list with a copied final user message is + written back to ``request_body``. + """ + schema = output_format.get("schema") + if not schema: + return + + messages = request_body.get("messages") + if not isinstance(messages, list) or not messages: + return + + last_user_idx = None + for i in range(len(messages) - 1, -1, -1): + message = messages[i] + if isinstance(message, dict) and message.get("role") == "user": + last_user_idx = i + break + + if last_user_idx is None: + return + + original = messages[last_user_idx] + content = original.get("content", []) + schema_block = {"type": "text", "text": json.dumps(schema)} + if isinstance(content, str): + new_content = [{"type": "text", "text": content}, schema_block] + elif isinstance(content, list): + new_content = [*content, schema_block] + else: + return + + new_messages = list(messages) + new_messages[last_user_idx] = {**original, "content": new_content} + request_body["messages"] = new_messages + + def remove_custom_field_from_tools(request_body: dict) -> None: """ Remove ``custom`` field from each tool in the request body. @@ -603,6 +685,62 @@ def is_claude_4_5_on_bedrock(model: str) -> bool: return any(pattern in model_lower for pattern in claude_4_5_patterns) +def normalize_bedrock_opus_output_config_effort(model: str, output_config: Any) -> None: + """ + Normalize Anthropic ``output_config.effort`` values for Bedrock Opus ids. + + Bedrock's Claude Opus request validator can accept a narrower effort + vocabulary than Anthropic's compatibility surface. The Bedrock ceiling is + read from ``model_prices_and_context_window.json`` via + ``bedrock_output_config_effort_ceiling``. + + Mutates ``output_config`` in place so callers can accept Claude Code's + ``xhigh`` input without forwarding a provider-invalid value. + """ + if not isinstance(output_config, dict): + return + + effort = output_config.get("effort") + if effort not in _BEDROCK_OUTPUT_CONFIG_EFFORT_ORDER: + return + + ceiling = _get_bedrock_output_config_effort_ceiling(model) + if ceiling is None: + return + + if ( + _BEDROCK_OUTPUT_CONFIG_EFFORT_ORDER[effort] + > _BEDROCK_OUTPUT_CONFIG_EFFORT_ORDER[ceiling] + ): + output_config["effort"] = ceiling + + +def _get_bedrock_output_config_effort_ceiling( + model: str, +) -> Optional[BedrockOutputConfigEffort]: + try: + model_info = get_cached_model_info()( + model=model, + custom_llm_provider="bedrock", + ) + except Exception: + return None + + ceiling = model_info.get("bedrock_output_config_effort_ceiling") + if isinstance(ceiling, str) and ceiling in _BEDROCK_OUTPUT_CONFIG_EFFORT_ORDER: + return ceiling # type: ignore[return-value] + + model_cost_key = model_info.get("key") + if not isinstance(model_cost_key, str): + return None + + local_model_info = _get_local_model_cost_map().get(model_cost_key, {}) + ceiling = local_model_info.get("bedrock_output_config_effort_ceiling") + if isinstance(ceiling, str) and ceiling in _BEDROCK_OUTPUT_CONFIG_EFFORT_ORDER: + return ceiling # type: ignore[return-value] + return None + + # Import after standalone functions to avoid circular imports from litellm.llms.bedrock.count_tokens.bedrock_token_counter import BedrockTokenCounter diff --git a/litellm/llms/bedrock/messages/invoke_transformations/anthropic_claude3_transformation.py b/litellm/llms/bedrock/messages/invoke_transformations/anthropic_claude3_transformation.py index 69b61298d3..b223f4534f 100644 --- a/litellm/llms/bedrock/messages/invoke_transformations/anthropic_claude3_transformation.py +++ b/litellm/llms/bedrock/messages/invoke_transformations/anthropic_claude3_transformation.py @@ -32,10 +32,13 @@ from litellm.llms.bedrock.chat.invoke_transformations.base_invoke_transformation AmazonInvokeConfig, ) from litellm.llms.bedrock.common_utils import ( + convert_bedrock_invoke_output_format_to_inline_schema, ensure_bedrock_anthropic_messages_tool_names, get_anthropic_beta_from_headers, is_claude_4_5_on_bedrock, + normalize_bedrock_opus_output_config_effort, normalize_tool_input_schema_types_for_bedrock_invoke, + pop_bedrock_invoke_output_config_format, remove_custom_field_from_tools, ) from litellm.types.llms.anthropic import ANTHROPIC_TOOL_SEARCH_BETA_HEADER @@ -450,145 +453,15 @@ class AmazonAnthropicClaudeMessagesConfig( else: anthropic_messages_request.pop("context_management", None) - def _convert_output_format_to_inline_schema( - self, - output_format: Dict, - anthropic_messages_request: Dict, - ) -> None: - """ - Convert Anthropic output_format to inline schema in message content. - - Bedrock Invoke doesn't support the output_format parameter, so we embed - the schema directly into the user message content as text instructions. - - This approach adds the schema to the last user message, instructing the model - to respond in the specified JSON format. - - Args: - output_format: The output_format dict with 'type' and 'schema' - anthropic_messages_request: The request dict to modify in-place - - Ref: https://aws.amazon.com/blogs/machine-learning/structured-data-response-with-amazon-bedrock-prompt-engineering-and-tool-use/ - """ - import json - - # Extract schema from output_format - schema = output_format.get("schema") - if not schema: - return - - # Get messages from the request - messages = anthropic_messages_request.get("messages", []) - if not messages: - return - - # Find the last user message - last_user_message_idx = None - for idx in range(len(messages) - 1, -1, -1): - if messages[idx].get("role") == "user": - last_user_message_idx = idx - break - - if last_user_message_idx is None: - return - - last_user_message = messages[last_user_message_idx] - content = last_user_message.get("content", []) - - # Ensure content is a list - if isinstance(content, str): - content = [{"type": "text", "text": content}] - last_user_message["content"] = content - - # Add schema as text content to the message - schema_text = {"type": "text", "text": json.dumps(schema)} - content.append(schema_text) - - def transform_anthropic_messages_request( + def _get_bedrock_invoke_anthropic_beta_headers( self, model: str, messages: List[Dict], anthropic_messages_optional_request_params: Dict, - litellm_params: GenericLiteLLMParams, headers: dict, - ) -> Dict: - anthropic_messages_request = AnthropicMessagesConfig.transform_anthropic_messages_request( - self=self, - model=model, - messages=messages, - anthropic_messages_optional_request_params=anthropic_messages_optional_request_params, - litellm_params=litellm_params, - headers=headers, - ) - ######################################################### - ############## BEDROCK Invoke SPECIFIC TRANSFORMATION ### - ######################################################### - - # 1. anthropic_version is required for all claude models - if "anthropic_version" not in anthropic_messages_request: - anthropic_messages_request["anthropic_version"] = ( - self.DEFAULT_BEDROCK_ANTHROPIC_API_VERSION - ) - - # 2. `stream` is not allowed in request body for bedrock invoke - if "stream" in anthropic_messages_request: - anthropic_messages_request.pop("stream", None) - - # 3. `model` is not allowed in request body for bedrock invoke - if "model" in anthropic_messages_request: - anthropic_messages_request.pop("model", None) - - injected_thinking_for_clear_thinking = ( - self._ensure_thinking_for_clear_thinking_context_management( - anthropic_messages_request=anthropic_messages_request, - model=model, - ) - ) - - # 4. Remove `ttl` field from cache_control in messages (Bedrock doesn't support it for older models) - self._remove_ttl_from_cache_control( - anthropic_messages_request=anthropic_messages_request, model=model - ) - - # 5. Convert `output_format` to inline schema (Bedrock invoke doesn't support output_format) - output_format = anthropic_messages_request.pop("output_format", None) - if output_format: - self._convert_output_format_to_inline_schema( - output_format=output_format, - anthropic_messages_request=anthropic_messages_request, - ) - - # 5a. Bedrock Invoke supports output_config (effort) for Claude 4.6+ models, - # but older models do not — strip it to avoid request rejection. - # Ref: https://github.com/BerriAI/litellm/issues/22797 - if not ( - _supports_factory( - model=model, - custom_llm_provider="bedrock", - key="supports_output_config", - ) - or AnthropicConfig._model_supports_effort_param(model) - ): - if anthropic_messages_request.pop("output_config", None) is not None: - verbose_logger.warning( - "Bedrock Invoke: stripping unsupported `output_config` for " - "model=%s — neither `supports_output_config` nor any " - "`supports_*_reasoning_effort` flag is set in " - "model_prices_and_context_window.json. Add the capability " - "flag to the model JSON entry if this model accepts " - "`output_config`.", - model, - ) - - # 5b. Remove `custom` field from tools (Bedrock doesn't support it) - # Claude Code sends `custom: {defer_loading: true}` on tool definitions, - # which causes Bedrock to reject the request with "Extra inputs are not permitted" - # Ref: https://github.com/BerriAI/litellm/issues/22847 - remove_custom_field_from_tools(anthropic_messages_request) - normalize_tool_input_schema_types_for_bedrock_invoke(anthropic_messages_request) - ensure_bedrock_anthropic_messages_tool_names(anthropic_messages_request) - - # 6. AUTO-INJECT beta headers based on features used + anthropic_messages_request: Dict, + injected_thinking_for_clear_thinking: bool, + ) -> List[str]: anthropic_model_info = AnthropicModelInfo() tools = anthropic_messages_optional_request_params.get("tools") messages_typed = cast(List[AllMessageValues], messages) @@ -651,6 +524,160 @@ class AmazonAnthropicClaudeMessagesConfig( dropped_user_betas, ) + return filtered_betas + + def _strip_unsupported_bedrock_invoke_fields( + self, + anthropic_messages_request: Dict, + ) -> Dict: + allowed = self.BEDROCK_INVOKE_ALLOWED_TOP_LEVEL_FIELDS + stripped = sorted(k for k in anthropic_messages_request if k not in allowed) + if stripped: + verbose_logger.debug( + "Bedrock Invoke: stripping unsupported top-level request fields: %s", + stripped, + ) + return {k: v for k, v in anthropic_messages_request.items() if k in allowed} + + @staticmethod + def _clamp_adaptive_reasoning_effort_for_bedrock( + model: str, optional_params: Dict + ) -> None: + """Lower ``reasoning_effort`` to the Bedrock effort ceiling before validation. + + The shared ``/v1/messages`` effort gate rejects tiers a model does not + natively support (e.g. ``xhigh`` on Opus 4.6). Bedrock's chat paths instead + clamp the tier to the model's ``bedrock_output_config_effort_ceiling`` so + Claude Code "goal mode" keeps working; mirror that here so the messages + path degrades ``xhigh`` -> ``max`` rather than 400-ing. Non-adaptive models + and models without a ceiling are left untouched. + """ + if not AnthropicModelInfo._is_adaptive_thinking_model(model): + return + effort = optional_params.get("reasoning_effort") + if not isinstance(effort, str): + return + clamped = {"effort": effort} + normalize_bedrock_opus_output_config_effort(model=model, output_config=clamped) + optional_params["reasoning_effort"] = clamped["effort"] + + def transform_anthropic_messages_request( + self, + model: str, + messages: List[Dict], + anthropic_messages_optional_request_params: Dict, + litellm_params: GenericLiteLLMParams, + headers: dict, + ) -> Dict: + self._clamp_adaptive_reasoning_effort_for_bedrock( + model=model, + optional_params=anthropic_messages_optional_request_params, + ) + anthropic_messages_request = AnthropicMessagesConfig.transform_anthropic_messages_request( + self=self, + model=model, + messages=messages, + anthropic_messages_optional_request_params=anthropic_messages_optional_request_params, + litellm_params=litellm_params, + headers=headers, + ) + ######################################################### + ############## BEDROCK Invoke SPECIFIC TRANSFORMATION ### + ######################################################### + + # 1. anthropic_version is required for all claude models + if "anthropic_version" not in anthropic_messages_request: + anthropic_messages_request["anthropic_version"] = ( + self.DEFAULT_BEDROCK_ANTHROPIC_API_VERSION + ) + + # 2. `stream` is not allowed in request body for bedrock invoke + if "stream" in anthropic_messages_request: + anthropic_messages_request.pop("stream", None) + + # 3. `model` is not allowed in request body for bedrock invoke + if "model" in anthropic_messages_request: + anthropic_messages_request.pop("model", None) + + injected_thinking_for_clear_thinking = ( + self._ensure_thinking_for_clear_thinking_context_management( + anthropic_messages_request=anthropic_messages_request, + model=model, + ) + ) + + # 4. Remove `ttl` field from cache_control in messages (Bedrock doesn't support it for older models) + self._remove_ttl_from_cache_control( + anthropic_messages_request=anthropic_messages_request, model=model + ) + + # 5. Convert structured-output params to inline schema. + # Bedrock Invoke doesn't support top-level `output_format`; its + # accepted `output_config` subset is also narrower than Anthropic's, so + # consume the newer `output_config.format` shape here instead of + # forwarding it as an unknown nested key. + existing_output_config = anthropic_messages_request.get("output_config") + if isinstance(existing_output_config, dict): + anthropic_messages_request["output_config"] = dict(existing_output_config) + output_format = anthropic_messages_request.pop("output_format", None) + output_config_format = pop_bedrock_invoke_output_config_format( + anthropic_messages_request + ) + if output_format: + convert_bedrock_invoke_output_format_to_inline_schema( + output_format=output_format, + request_body=anthropic_messages_request, + ) + elif output_config_format: + convert_bedrock_invoke_output_format_to_inline_schema( + output_format=output_config_format, + request_body=anthropic_messages_request, + ) + normalize_bedrock_opus_output_config_effort( + model=model, + output_config=anthropic_messages_request.get("output_config"), + ) + + # 5a. Bedrock Invoke supports output_config (effort) for Claude 4.6+ models, + # but older models do not — strip it to avoid request rejection. + # Ref: https://github.com/BerriAI/litellm/issues/22797 + if not ( + _supports_factory( + model=model, + custom_llm_provider="bedrock", + key="supports_output_config", + ) + or AnthropicConfig._model_supports_effort_param(model) + ): + if anthropic_messages_request.pop("output_config", None) is not None: + verbose_logger.warning( + "Bedrock Invoke: stripping unsupported `output_config` for " + "model=%s — neither `supports_output_config` nor any " + "`supports_*_reasoning_effort` flag is set in " + "model_prices_and_context_window.json. Add the capability " + "flag to the model JSON entry if this model accepts " + "`output_config`.", + model, + ) + + # 5b. Remove `custom` field from tools (Bedrock doesn't support it) + # Claude Code sends `custom: {defer_loading: true}` on tool definitions, + # which causes Bedrock to reject the request with "Extra inputs are not permitted" + # Ref: https://github.com/BerriAI/litellm/issues/22847 + remove_custom_field_from_tools(anthropic_messages_request) + normalize_tool_input_schema_types_for_bedrock_invoke(anthropic_messages_request) + ensure_bedrock_anthropic_messages_tool_names(anthropic_messages_request) + + # 6. AUTO-INJECT beta headers based on features used + filtered_betas = self._get_bedrock_invoke_anthropic_beta_headers( + model=model, + messages=messages, + anthropic_messages_optional_request_params=anthropic_messages_optional_request_params, + headers=headers, + anthropic_messages_request=anthropic_messages_request, + injected_thinking_for_clear_thinking=injected_thinking_for_clear_thinking, + ) + if filtered_betas: anthropic_messages_request["anthropic_beta"] = filtered_betas @@ -669,16 +696,9 @@ class AmazonAnthropicClaudeMessagesConfig( # Catches Anthropic-only extensions (output_config, speed, mcp_servers, ...) # and any future additions Claude Code may start sending. ``context_management`` # has already been pre-filtered to its Bedrock-supported subset above. - allowed = self.BEDROCK_INVOKE_ALLOWED_TOP_LEVEL_FIELDS - stripped = sorted(k for k in anthropic_messages_request if k not in allowed) - if stripped: - verbose_logger.debug( - "Bedrock Invoke: stripping unsupported top-level request fields: %s", - stripped, - ) - anthropic_messages_request = { - k: v for k, v in anthropic_messages_request.items() if k in allowed - } + anthropic_messages_request = self._strip_unsupported_bedrock_invoke_fields( + anthropic_messages_request + ) return anthropic_messages_request diff --git a/litellm/model_prices_and_context_window_backup.json b/litellm/model_prices_and_context_window_backup.json index ec16f19799..62e0f6c4c3 100644 --- a/litellm/model_prices_and_context_window_backup.json +++ b/litellm/model_prices_and_context_window_backup.json @@ -982,7 +982,9 @@ "supports_tool_choice": true, "supports_vision": true, "tool_use_system_prompt_tokens": 159, - "supports_native_structured_output": true + "supports_native_structured_output": true, + "supports_output_config": true, + "bedrock_output_config_effort_ceiling": "high" }, "anthropic.claude-opus-4-6-v1": { "cache_creation_input_token_cost": 6.25e-06, @@ -1013,7 +1015,8 @@ "supports_native_structured_output": true, "supports_output_config": true, "supports_max_reasoning_effort": true, - "supports_minimal_reasoning_effort": true + "supports_minimal_reasoning_effort": true, + "bedrock_output_config_effort_ceiling": "max" }, "global.anthropic.claude-opus-4-6-v1": { "cache_creation_input_token_cost": 6.25e-06, @@ -1044,7 +1047,8 @@ "supports_native_structured_output": true, "supports_output_config": true, "supports_max_reasoning_effort": true, - "supports_minimal_reasoning_effort": true + "supports_minimal_reasoning_effort": true, + "bedrock_output_config_effort_ceiling": "max" }, "us.anthropic.claude-opus-4-6-v1": { "cache_creation_input_token_cost": 6.875e-06, @@ -1075,7 +1079,8 @@ "supports_native_structured_output": true, "supports_output_config": true, "supports_max_reasoning_effort": true, - "supports_minimal_reasoning_effort": true + "supports_minimal_reasoning_effort": true, + "bedrock_output_config_effort_ceiling": "max" }, "eu.anthropic.claude-opus-4-6-v1": { "cache_creation_input_token_cost": 6.875e-06, @@ -1105,7 +1110,8 @@ "supports_native_structured_output": true, "supports_output_config": true, "supports_max_reasoning_effort": true, - "supports_minimal_reasoning_effort": true + "supports_minimal_reasoning_effort": true, + "bedrock_output_config_effort_ceiling": "max" }, "au.anthropic.claude-opus-4-6-v1": { "cache_creation_input_token_cost": 6.875e-06, @@ -1135,7 +1141,8 @@ "supports_native_structured_output": true, "supports_output_config": true, "supports_max_reasoning_effort": true, - "supports_minimal_reasoning_effort": true + "supports_minimal_reasoning_effort": true, + "bedrock_output_config_effort_ceiling": "max" }, "anthropic.claude-opus-4-7": { "cache_creation_input_token_cost": 6.25e-06, @@ -1166,7 +1173,9 @@ "tool_use_system_prompt_tokens": 346, "supports_native_structured_output": true, "supports_max_reasoning_effort": true, - "supports_minimal_reasoning_effort": true + "supports_minimal_reasoning_effort": true, + "supports_output_config": true, + "bedrock_output_config_effort_ceiling": "xhigh" }, "anthropic.claude-mythos-preview": { "input_cost_per_token": 0, @@ -1212,7 +1221,9 @@ "tool_use_system_prompt_tokens": 346, "supports_native_structured_output": true, "supports_max_reasoning_effort": true, - "supports_minimal_reasoning_effort": true + "supports_minimal_reasoning_effort": true, + "supports_output_config": true, + "bedrock_output_config_effort_ceiling": "xhigh" }, "us.anthropic.claude-opus-4-7": { "cache_creation_input_token_cost": 6.875e-06, @@ -1243,7 +1254,9 @@ "tool_use_system_prompt_tokens": 346, "supports_native_structured_output": true, "supports_max_reasoning_effort": true, - "supports_minimal_reasoning_effort": true + "supports_minimal_reasoning_effort": true, + "supports_output_config": true, + "bedrock_output_config_effort_ceiling": "xhigh" }, "eu.anthropic.claude-opus-4-7": { "cache_creation_input_token_cost": 6.875e-06, @@ -1273,7 +1286,9 @@ "tool_use_system_prompt_tokens": 346, "supports_native_structured_output": true, "supports_max_reasoning_effort": true, - "supports_minimal_reasoning_effort": true + "supports_minimal_reasoning_effort": true, + "supports_output_config": true, + "bedrock_output_config_effort_ceiling": "xhigh" }, "au.anthropic.claude-opus-4-7": { "cache_creation_input_token_cost": 6.875e-06, @@ -1303,7 +1318,9 @@ "tool_use_system_prompt_tokens": 346, "supports_native_structured_output": true, "supports_max_reasoning_effort": true, - "supports_minimal_reasoning_effort": true + "supports_minimal_reasoning_effort": true, + "supports_output_config": true, + "bedrock_output_config_effort_ceiling": "xhigh" }, "anthropic.claude-sonnet-4-6": { "cache_creation_input_token_cost": 3.75e-06, @@ -9804,7 +9821,8 @@ "supports_response_schema": true, "supports_tool_choice": true, "supports_vision": true, - "tool_use_system_prompt_tokens": 159 + "tool_use_system_prompt_tokens": 159, + "supports_output_config": true }, "claude-opus-4-5": { "cache_creation_input_token_cost": 6.25e-06, @@ -9832,7 +9850,8 @@ "supports_response_schema": true, "supports_tool_choice": true, "supports_vision": true, - "tool_use_system_prompt_tokens": 159 + "tool_use_system_prompt_tokens": 159, + "supports_output_config": true }, "claude-opus-4-6": { "cache_creation_input_token_cost": 6.25e-06, @@ -31808,7 +31827,9 @@ "supports_tool_choice": true, "supports_vision": true, "tool_use_system_prompt_tokens": 159, - "supports_native_structured_output": true + "supports_native_structured_output": true, + "supports_output_config": true, + "bedrock_output_config_effort_ceiling": "high" }, "global.anthropic.claude-opus-4-5-20251101-v1:0": { "cache_creation_input_token_cost": 6.25e-06, @@ -31837,7 +31858,9 @@ "supports_tool_choice": true, "supports_vision": true, "tool_use_system_prompt_tokens": 159, - "supports_native_structured_output": true + "supports_native_structured_output": true, + "supports_output_config": true, + "bedrock_output_config_effort_ceiling": "high" }, "eu.anthropic.claude-opus-4-5-20251101-v1:0": { "cache_creation_input_token_cost": 6.25e-06, @@ -31865,7 +31888,9 @@ "supports_tool_choice": true, "supports_vision": true, "tool_use_system_prompt_tokens": 159, - "supports_native_structured_output": true + "supports_native_structured_output": true, + "supports_output_config": true, + "bedrock_output_config_effort_ceiling": "high" }, "us.anthropic.claude-sonnet-4-20250514-v1:0": { "cache_creation_input_token_cost": 3.75e-06, diff --git a/litellm/types/llms/anthropic.py b/litellm/types/llms/anthropic.py index 1c4d31d21a..bbb892a027 100644 --- a/litellm/types/llms/anthropic.py +++ b/litellm/types/llms/anthropic.py @@ -39,7 +39,8 @@ class AnthropicOutputSchema(TypedDict, total=False): class AnthropicOutputConfig(TypedDict, total=False): """Configuration for controlling Claude's output behavior.""" - effort: Literal["high", "medium", "low"] + effort: Literal["high", "medium", "low", "xhigh", "max"] + format: AnthropicOutputSchema class AnthropicMessagesTool(TypedDict, total=False): diff --git a/litellm/types/utils.py b/litellm/types/utils.py index e7bce27170..8f471b62b5 100644 --- a/litellm/types/utils.py +++ b/litellm/types/utils.py @@ -148,6 +148,9 @@ class ProviderSpecificModelInfo(TypedDict, total=False): supports_xhigh_reasoning_effort: Optional[bool] supports_max_reasoning_effort: Optional[bool] supports_output_config: Optional[bool] + bedrock_output_config_effort_ceiling: Optional[ + Literal["low", "medium", "high", "max", "xhigh"] + ] class SearchContextCostPerQuery(TypedDict, total=False): diff --git a/litellm/utils.py b/litellm/utils.py index 760a615664..5a9dccc089 100644 --- a/litellm/utils.py +++ b/litellm/utils.py @@ -6036,6 +6036,9 @@ def _get_model_info_helper( # noqa: PLR0915 supports_max_reasoning_effort=_model_info.get( "supports_max_reasoning_effort", None ), + bedrock_output_config_effort_ceiling=_model_info.get( + "bedrock_output_config_effort_ceiling", None + ), supports_computer_use=_model_info.get("supports_computer_use", None), search_context_cost_per_query=_model_info.get( "search_context_cost_per_query", None diff --git a/model_prices_and_context_window.json b/model_prices_and_context_window.json index 6e1c79c4e3..0689066e17 100644 --- a/model_prices_and_context_window.json +++ b/model_prices_and_context_window.json @@ -982,7 +982,9 @@ "supports_tool_choice": true, "supports_vision": true, "tool_use_system_prompt_tokens": 159, - "supports_native_structured_output": true + "supports_native_structured_output": true, + "supports_output_config": true, + "bedrock_output_config_effort_ceiling": "high" }, "anthropic.claude-opus-4-6-v1": { "cache_creation_input_token_cost": 6.25e-06, @@ -1013,7 +1015,8 @@ "supports_native_structured_output": true, "supports_output_config": true, "supports_max_reasoning_effort": true, - "supports_minimal_reasoning_effort": true + "supports_minimal_reasoning_effort": true, + "bedrock_output_config_effort_ceiling": "max" }, "global.anthropic.claude-opus-4-6-v1": { "cache_creation_input_token_cost": 6.25e-06, @@ -1044,7 +1047,8 @@ "supports_native_structured_output": true, "supports_output_config": true, "supports_max_reasoning_effort": true, - "supports_minimal_reasoning_effort": true + "supports_minimal_reasoning_effort": true, + "bedrock_output_config_effort_ceiling": "max" }, "us.anthropic.claude-opus-4-6-v1": { "cache_creation_input_token_cost": 6.875e-06, @@ -1075,7 +1079,8 @@ "supports_native_structured_output": true, "supports_output_config": true, "supports_max_reasoning_effort": true, - "supports_minimal_reasoning_effort": true + "supports_minimal_reasoning_effort": true, + "bedrock_output_config_effort_ceiling": "max" }, "eu.anthropic.claude-opus-4-6-v1": { "cache_creation_input_token_cost": 6.875e-06, @@ -1105,7 +1110,8 @@ "supports_native_structured_output": true, "supports_output_config": true, "supports_max_reasoning_effort": true, - "supports_minimal_reasoning_effort": true + "supports_minimal_reasoning_effort": true, + "bedrock_output_config_effort_ceiling": "max" }, "au.anthropic.claude-opus-4-6-v1": { "cache_creation_input_token_cost": 6.875e-06, @@ -1135,7 +1141,8 @@ "supports_native_structured_output": true, "supports_output_config": true, "supports_max_reasoning_effort": true, - "supports_minimal_reasoning_effort": true + "supports_minimal_reasoning_effort": true, + "bedrock_output_config_effort_ceiling": "max" }, "anthropic.claude-opus-4-7": { "cache_creation_input_token_cost": 6.25e-06, @@ -1166,7 +1173,9 @@ "tool_use_system_prompt_tokens": 346, "supports_native_structured_output": true, "supports_max_reasoning_effort": true, - "supports_minimal_reasoning_effort": true + "supports_minimal_reasoning_effort": true, + "supports_output_config": true, + "bedrock_output_config_effort_ceiling": "xhigh" }, "anthropic.claude-mythos-preview": { "input_cost_per_token": 0, @@ -1212,7 +1221,9 @@ "tool_use_system_prompt_tokens": 346, "supports_native_structured_output": true, "supports_max_reasoning_effort": true, - "supports_minimal_reasoning_effort": true + "supports_minimal_reasoning_effort": true, + "supports_output_config": true, + "bedrock_output_config_effort_ceiling": "xhigh" }, "us.anthropic.claude-opus-4-7": { "cache_creation_input_token_cost": 6.875e-06, @@ -1243,7 +1254,9 @@ "tool_use_system_prompt_tokens": 346, "supports_native_structured_output": true, "supports_max_reasoning_effort": true, - "supports_minimal_reasoning_effort": true + "supports_minimal_reasoning_effort": true, + "supports_output_config": true, + "bedrock_output_config_effort_ceiling": "xhigh" }, "eu.anthropic.claude-opus-4-7": { "cache_creation_input_token_cost": 6.875e-06, @@ -1273,7 +1286,9 @@ "tool_use_system_prompt_tokens": 346, "supports_native_structured_output": true, "supports_max_reasoning_effort": true, - "supports_minimal_reasoning_effort": true + "supports_minimal_reasoning_effort": true, + "supports_output_config": true, + "bedrock_output_config_effort_ceiling": "xhigh" }, "au.anthropic.claude-opus-4-7": { "cache_creation_input_token_cost": 6.875e-06, @@ -1303,7 +1318,9 @@ "tool_use_system_prompt_tokens": 346, "supports_native_structured_output": true, "supports_max_reasoning_effort": true, - "supports_minimal_reasoning_effort": true + "supports_minimal_reasoning_effort": true, + "supports_output_config": true, + "bedrock_output_config_effort_ceiling": "xhigh" }, "anthropic.claude-sonnet-4-6": { "cache_creation_input_token_cost": 3.75e-06, @@ -9804,7 +9821,8 @@ "supports_response_schema": true, "supports_tool_choice": true, "supports_vision": true, - "tool_use_system_prompt_tokens": 159 + "tool_use_system_prompt_tokens": 159, + "supports_output_config": true }, "claude-opus-4-5": { "cache_creation_input_token_cost": 6.25e-06, @@ -9832,7 +9850,8 @@ "supports_response_schema": true, "supports_tool_choice": true, "supports_vision": true, - "tool_use_system_prompt_tokens": 159 + "tool_use_system_prompt_tokens": 159, + "supports_output_config": true }, "claude-opus-4-6": { "cache_creation_input_token_cost": 6.25e-06, @@ -31683,7 +31702,9 @@ "supports_tool_choice": true, "supports_vision": true, "tool_use_system_prompt_tokens": 159, - "supports_native_structured_output": true + "supports_native_structured_output": true, + "supports_output_config": true, + "bedrock_output_config_effort_ceiling": "high" }, "global.anthropic.claude-opus-4-5-20251101-v1:0": { "cache_creation_input_token_cost": 6.25e-06, @@ -31712,7 +31733,9 @@ "supports_tool_choice": true, "supports_vision": true, "tool_use_system_prompt_tokens": 159, - "supports_native_structured_output": true + "supports_native_structured_output": true, + "supports_output_config": true, + "bedrock_output_config_effort_ceiling": "high" }, "eu.anthropic.claude-opus-4-5-20251101-v1:0": { "cache_creation_input_token_cost": 6.25e-06, @@ -31740,7 +31763,9 @@ "supports_tool_choice": true, "supports_vision": true, "tool_use_system_prompt_tokens": 159, - "supports_native_structured_output": true + "supports_native_structured_output": true, + "supports_output_config": true, + "bedrock_output_config_effort_ceiling": "high" }, "us.anthropic.claude-sonnet-4-20250514-v1:0": { "cache_creation_input_token_cost": 3.75e-06, diff --git a/tests/llm_translation/reasoning_effort_grid/grid_spec.py b/tests/llm_translation/reasoning_effort_grid/grid_spec.py index 993643e0fc..2f9735274c 100644 --- a/tests/llm_translation/reasoning_effort_grid/grid_spec.py +++ b/tests/llm_translation/reasoning_effort_grid/grid_spec.py @@ -22,6 +22,7 @@ class ModelEntry: required_env: FrozenSet[str] = field(default_factory=frozenset) caps: FrozenSet[str] = field(default_factory=frozenset) fail_reason: Optional[str] = None + bedrock_effort_ceiling: Optional[str] = None def params(self) -> Dict[str, str]: return dict(self.extra_params) @@ -59,9 +60,31 @@ _ADAPTIVE_EFFORT_LABEL: Dict[str, str] = { "max": "max", } +_EFFORT_RANK: Dict[str, int] = { + "low": 0, + "medium": 1, + "high": 2, + "max": 3, + "xhigh": 4, +} + _BAD_REQUEST_EFFORTS: FrozenSet[str] = frozenset({"disabled", "invalid", ""}) +def _bedrock_clamps_effort(model: "ModelEntry", effort: str) -> bool: + """Whether Bedrock will clamp ``effort`` down to ``bedrock_effort_ceiling``. + + Bedrock chat/messages paths clamp unsupported high tiers (e.g. ``xhigh`` + on Opus 4.6) to the model's ceiling rather than rejecting them, so the + missing native capability is OK — the wire effort just degrades. + """ + if model.bedrock_effort_ceiling is None: + return False + if effort not in _EFFORT_RANK or model.bedrock_effort_ceiling not in _EFFORT_RANK: + return False + return _EFFORT_RANK[effort] > _EFFORT_RANK[model.bedrock_effort_ceiling] + + def expected(model: ModelEntry, effort: str) -> CellExpectation: if effort in ("__omit__", "none"): if model.mode == "budget": @@ -73,14 +96,20 @@ def expected(model: ModelEntry, effort: str) -> CellExpectation: if effort in ("xhigh", "max"): cap = f"supports_{effort}_reasoning_effort" - if cap not in model.caps: + if cap not in model.caps and not _bedrock_clamps_effort(model, effort): return CellExpectation(status=400, thinking_type=OMIT) if model.mode == "adaptive": + wire_effort = _ADAPTIVE_EFFORT_LABEL[effort] + if model.bedrock_effort_ceiling is not None: + wire_rank = _EFFORT_RANK[wire_effort] + ceiling_rank = _EFFORT_RANK[model.bedrock_effort_ceiling] + if wire_rank > ceiling_rank: + wire_effort = model.bedrock_effort_ceiling return CellExpectation( status=200, thinking_type="adaptive", - output_config_effort=_ADAPTIVE_EFFORT_LABEL[effort], + output_config_effort=wire_effort, ) return CellExpectation( @@ -219,6 +248,7 @@ BEDROCK_CONVERSE_MODELS: Tuple[ModelEntry, ...] = ( extra_params=(("aws_region_name", "us-east-1"),), required_env=_BEDROCK_REQ, caps=_CAPS_4_6, + bedrock_effort_ceiling="max", ), ModelEntry( alias="bedrock-claude-sonnet-4-6", @@ -247,6 +277,7 @@ BEDROCK_INVOKE_CHAT_MODELS: Tuple[ModelEntry, ...] = ( extra_params=(("aws_region_name", "us-east-1"),), required_env=_BEDROCK_REQ, caps=_CAPS_4_6, + bedrock_effort_ceiling="max", ), ModelEntry( alias="bedrock-invoke-claude-sonnet-4-6", diff --git a/tests/test_litellm/llms/anthropic/chat/test_anthropic_chat_transformation.py b/tests/test_litellm/llms/anthropic/chat/test_anthropic_chat_transformation.py index 7d9e476830..687c5a2e73 100644 --- a/tests/test_litellm/llms/anthropic/chat/test_anthropic_chat_transformation.py +++ b/tests/test_litellm/llms/anthropic/chat/test_anthropic_chat_transformation.py @@ -1622,6 +1622,29 @@ def test_effort_output_config_preservation(): assert result["output_config"]["effort"] == "medium" +def test_output_config_format_preservation_and_beta_header(): + """Test that output_config.format is preserved and treated as structured output.""" + config = AnthropicConfig() + output_format = { + "type": "json_schema", + "schema": {"type": "object", "properties": {"answer": {"type": "string"}}}, + } + optional_params = {"output_config": {"format": output_format, "effort": "xhigh"}} + + result = config.transform_request( + model="claude-opus-4-7", + messages=[{"role": "user", "content": "Test"}], + optional_params=optional_params, + litellm_params={}, + headers={}, + ) + headers = config.update_headers_with_optional_anthropic_beta({}, optional_params) + + assert result["output_config"]["format"] == output_format + assert result["output_config"]["effort"] == "xhigh" + assert "structured-outputs-2025-11-13" in headers["anthropic-beta"] + + def test_effort_beta_header_injection(): """Test that effort beta header is automatically added when output_config is detected.""" from litellm.llms.anthropic.common_utils import AnthropicModelInfo @@ -1648,7 +1671,7 @@ def test_effort_validation(): messages = [{"role": "user", "content": "Test"}] - # Valid values should work + # Valid values should work (xhigh is Opus 4.7+ only, not 4.5) for effort in ["high", "medium", "low"]: optional_params = {"output_config": {"effort": effort}} result = config.transform_request( @@ -2513,14 +2536,14 @@ def test_reasoning_effort_accepts_dict_shape_for_adaptive_model(reasoning_effort ) # thinking must be set (adaptive for 4.6+) - assert "thinking" in result, ( - f"thinking missing for reasoning_effort={reasoning_effort_value!r}" - ) + assert ( + "thinking" in result + ), f"thinking missing for reasoning_effort={reasoning_effort_value!r}" assert result["thinking"]["type"] == "adaptive" # output_config must carry the mapped effort - assert "output_config" in result, ( - f"output_config missing for reasoning_effort={reasoning_effort_value!r}" - ) + assert ( + "output_config" in result + ), f"output_config missing for reasoning_effort={reasoning_effort_value!r}" assert result["output_config"]["effort"] == "low" @@ -2532,7 +2555,9 @@ def test_reasoning_effort_accepts_dict_shape_for_adaptive_model(reasoning_effort {"effort": "low", "summary": "concise"}, ], ) -def test_reasoning_effort_accepts_dict_shape_for_non_adaptive_model(reasoning_effort_value): +def test_reasoning_effort_accepts_dict_shape_for_non_adaptive_model( + reasoning_effort_value, +): """ Non-adaptive (pre-4.6) branch: dict-shape reasoning_effort must still map to ``thinking.type='enabled'`` + ``budget_tokens``. ``output_config`` must @@ -2547,9 +2572,9 @@ def test_reasoning_effort_accepts_dict_shape_for_non_adaptive_model(reasoning_ef drop_params=False, ) - assert "thinking" in result, ( - f"thinking missing for reasoning_effort={reasoning_effort_value!r}" - ) + assert ( + "thinking" in result + ), f"thinking missing for reasoning_effort={reasoning_effort_value!r}" assert result["thinking"]["type"] == "enabled" assert "budget_tokens" in result["thinking"] assert result["thinking"]["budget_tokens"] > 0 @@ -2582,12 +2607,12 @@ def test_reasoning_effort_unparseable_dict_is_dropped(bad_value): model="claude-sonnet-4-6-20260219", drop_params=False, ) - assert "thinking" not in result, ( - f"thinking should not be set for bad value {bad_value!r}" - ) - assert "output_config" not in result, ( - f"output_config should not be set for bad value {bad_value!r}" - ) + assert ( + "thinking" not in result + ), f"thinking should not be set for bad value {bad_value!r}" + assert ( + "output_config" not in result + ), f"output_config should not be set for bad value {bad_value!r}" @pytest.mark.parametrize( diff --git a/tests/test_litellm/llms/anthropic/experimental_pass_through/messages/test_anthropic_messages_structured_outputs.py b/tests/test_litellm/llms/anthropic/experimental_pass_through/messages/test_anthropic_messages_structured_outputs.py index 3c81bfaa0f..e6d5c6f4ee 100644 --- a/tests/test_litellm/llms/anthropic/experimental_pass_through/messages/test_anthropic_messages_structured_outputs.py +++ b/tests/test_litellm/llms/anthropic/experimental_pass_through/messages/test_anthropic_messages_structured_outputs.py @@ -48,6 +48,39 @@ def test_output_format_supported_and_transforms_correctly(): assert "structured-outputs-2025-11-13" in headers["anthropic-beta"] +def test_output_config_format_supported_and_transforms_correctly(): + """Test that output_config.format is preserved and adds the structured-output beta.""" + config = AnthropicMessagesConfig() + + supported_params = config.get_supported_anthropic_messages_params("claude-opus-4-7") + assert "output_config" in supported_params + + output_format = { + "type": "json_schema", + "schema": {"type": "object", "properties": {"result": {"type": "string"}}}, + } + optional_params = { + "max_tokens": 1024, + "output_config": {"format": output_format, "effort": "xhigh"}, + } + headers = {} + + result = config.transform_anthropic_messages_request( + model="claude-opus-4-7", + messages=[{"role": "user", "content": "test"}], + anthropic_messages_optional_request_params=optional_params.copy(), + litellm_params={}, + headers=headers, + ) + + headers = config._update_headers_with_anthropic_beta(headers, optional_params) + + assert result["output_config"]["format"] == output_format + assert result["output_config"]["effort"] == "xhigh" + assert "anthropic-beta" in headers + assert "structured-outputs-2025-11-13" in headers["anthropic-beta"] + + def test_output_format_works_with_bedrock_and_azure(): """Test that output_format works with Bedrock and Azure Foundry models.""" config = AnthropicMessagesConfig() diff --git a/tests/test_litellm/llms/anthropic/experimental_pass_through/messages/test_reasoning_effort_translation.py b/tests/test_litellm/llms/anthropic/experimental_pass_through/messages/test_reasoning_effort_translation.py index 83716b8c8d..54bf0c4ac0 100644 --- a/tests/test_litellm/llms/anthropic/experimental_pass_through/messages/test_reasoning_effort_translation.py +++ b/tests/test_litellm/llms/anthropic/experimental_pass_through/messages/test_reasoning_effort_translation.py @@ -6,6 +6,9 @@ from litellm.llms.anthropic.common_utils import AnthropicError from litellm.llms.anthropic.experimental_pass_through.messages.transformation import ( AnthropicMessagesConfig, ) +from litellm.llms.bedrock.messages.invoke_transformations.anthropic_claude3_transformation import ( + AmazonAnthropicClaudeMessagesConfig, +) @pytest.mark.parametrize( @@ -102,7 +105,6 @@ def test_invalid_reasoning_effort_raises_400(bad_effort): "model,bad_effort", [ ("claude-opus-4-6", "xhigh"), - ("bedrock/invoke/us.anthropic.claude-opus-4-6-v1", "xhigh"), ("claude-sonnet-4-6", "xhigh"), ], ) @@ -123,6 +125,56 @@ def test_reasoning_effort_unsupported_tier_raises_400_messages(model, bad_effort assert "not supported by this model" in str(exc_info.value) +@pytest.mark.parametrize( + "model,effort,expected_effort", + [ + ("invoke/us.anthropic.claude-opus-4-6-v1", "xhigh", "max"), + ("invoke/us.anthropic.claude-opus-4-6-v1", "max", "max"), + ("invoke/us.anthropic.claude-opus-4-6-v1", "high", "high"), + ("invoke/us.anthropic.claude-opus-4-7", "xhigh", "xhigh"), + ], +) +def test_bedrock_invoke_messages_clamps_effort_to_ceiling( + model, effort, expected_effort +): + """Bedrock Invoke /v1/messages degrades effort to the model's ceiling. + + Claude Code "goal mode" sends ``xhigh``; Opus 4.6 must clamp to ``max`` + instead of raising, while Opus 4.7 (ceiling ``xhigh``) keeps ``xhigh``. + """ + config = AmazonAnthropicClaudeMessagesConfig() + optional_params = {"max_tokens": 1024, "reasoning_effort": effort} + + result = config.transform_anthropic_messages_request( + model=model, + messages=[{"role": "user", "content": "Hello"}], + anthropic_messages_optional_request_params=optional_params, + litellm_params={}, + headers={}, + ) + + assert result["output_config"]["effort"] == expected_effort + assert result["thinking"]["type"] == "adaptive" + + +def test_bedrock_invoke_messages_rejects_xhigh_without_ceiling(): + """Sonnet 4.6 on Bedrock has no effort ceiling, so xhigh is still rejected.""" + config = AmazonAnthropicClaudeMessagesConfig() + optional_params = {"max_tokens": 1024, "reasoning_effort": "xhigh"} + + with pytest.raises(AnthropicError) as exc_info: + config.transform_anthropic_messages_request( + model="invoke/us.anthropic.claude-sonnet-4-6", + messages=[{"role": "user", "content": "Hello"}], + anthropic_messages_optional_request_params=optional_params, + litellm_params={}, + headers={}, + ) + + assert exc_info.value.status_code == 400 + assert "not supported by this model" in str(exc_info.value) + + @pytest.mark.parametrize( "model", [ diff --git a/tests/test_litellm/llms/bedrock/chat/invoke_transformations/test_bedrock_chat_invoke_transformations_anthropic_claude3_transformation.py b/tests/test_litellm/llms/bedrock/chat/invoke_transformations/test_bedrock_chat_invoke_transformations_anthropic_claude3_transformation.py index b2e254901f..4c4c0e17a3 100644 --- a/tests/test_litellm/llms/bedrock/chat/invoke_transformations/test_bedrock_chat_invoke_transformations_anthropic_claude3_transformation.py +++ b/tests/test_litellm/llms/bedrock/chat/invoke_transformations/test_bedrock_chat_invoke_transformations_anthropic_claude3_transformation.py @@ -430,6 +430,61 @@ def test_output_config_forwarded_for_bedrock_chat_invoke_request(): assert result["max_tokens"] == 100 +def test_output_config_format_converted_for_bedrock_chat_invoke_request(): + """Bedrock Invoke chat path consumes ``output_config.format`` before forwarding.""" + config = AmazonAnthropicClaudeConfig() + schema = { + "type": "object", + "properties": {"answer": {"type": "string"}}, + } + + result = config.transform_request( + model="anthropic.claude-opus-4-7", + messages=[{"role": "user", "content": "test"}], + optional_params={ + "max_tokens": 100, + "output_config": { + "effort": "xhigh", + "format": {"type": "json_schema", "schema": schema}, + }, + }, + litellm_params={}, + headers={}, + ) + + assert result.get("output_config") == {"effort": "xhigh"} + last_content = result["messages"][0]["content"] + assert json.loads(last_content[-1]["text"]) == schema + + +@pytest.mark.parametrize( + "model,expected_effort", + [ + ("anthropic.claude-opus-4-5-20251101-v1:0", "high"), + ("anthropic.claude-opus-4-6-v1", "max"), + ("anthropic.claude-opus-4-7", "xhigh"), + ], +) +def test_output_config_effort_normalized_for_bedrock_chat_invoke_request( + model, expected_effort +): + """Bedrock Invoke chat path accepts ``xhigh`` and forwards the provider-safe effort.""" + config = AmazonAnthropicClaudeConfig() + + result = config.transform_request( + model=model, + messages=[{"role": "user", "content": "test"}], + optional_params={ + "max_tokens": 100, + "output_config": {"effort": "xhigh"}, + }, + litellm_params={}, + headers={}, + ) + + assert result.get("output_config") == {"effort": expected_effort} + + def test_bedrock_chat_invoke_checks_output_config_support_with_bedrock_provider(): config = AmazonAnthropicClaudeConfig() messages = [{"role": "user", "content": "test"}] diff --git a/tests/test_litellm/llms/bedrock/chat/test_converse_transformation.py b/tests/test_litellm/llms/bedrock/chat/test_converse_transformation.py index 5f2ed3dc00..c8e72b7ac5 100644 --- a/tests/test_litellm/llms/bedrock/chat/test_converse_transformation.py +++ b/tests/test_litellm/llms/bedrock/chat/test_converse_transformation.py @@ -318,6 +318,7 @@ def test_reasoning_effort_none_omits_thinking_for_anthropic_converse(model): ("bedrock/converse/us.anthropic.claude-opus-4-7", "high", "high"), ("bedrock/converse/us.anthropic.claude-opus-4-7", "xhigh", "xhigh"), ("bedrock/converse/us.anthropic.claude-opus-4-7", "max", "max"), + ("bedrock/converse/us.anthropic.claude-opus-4-6-v1", "xhigh", "max"), ("bedrock/converse/us.anthropic.claude-opus-4-6-v1", "max", "max"), ("bedrock/converse/us.anthropic.claude-sonnet-4-6", "high", "high"), ("bedrock/converse/us.anthropic.claude-sonnet-4-6", "minimal", "low"), @@ -369,6 +370,132 @@ def test_output_config_effort_forwarded_into_additional_request_fields(model): assert additional.get("output_config") == {"effort": "high"} +def test_output_config_format_translated_to_native_output_config_converse(): + """``output_config.format`` becomes Bedrock ``outputConfig`` and is not forwarded raw.""" + config = AmazonConverseConfig() + schema = { + "type": "object", + "properties": {"answer": {"type": "string"}}, + } + + result = config._transform_request( + model="bedrock/converse/us.anthropic.claude-opus-4-7", + messages=[{"role": "user", "content": "hi"}], + optional_params={ + "maxTokens": 256, + "thinking": {"type": "adaptive"}, + "output_config": { + "effort": "xhigh", + "format": {"type": "json_schema", "schema": schema}, + }, + }, + litellm_params={}, + headers={}, + ) + + additional = result.get("additionalModelRequestFields", {}) + assert additional.get("output_config") == {"effort": "xhigh"} + assert "format" not in additional["output_config"] + assert result["outputConfig"]["textFormat"]["type"] == "json_schema" + parsed_schema = json.loads( + result["outputConfig"]["textFormat"]["structure"]["jsonSchema"]["schema"] + ) + assert parsed_schema == {**schema, "additionalProperties": False} + + +def test_output_config_format_dropped_on_unsupported_converse_model_warns(caplog): + """When Converse model lacks native structured-output support, the silently + dropped ``output_config.format`` must surface as a warning so callers can + diagnose plain-text responses.""" + from unittest.mock import patch + + config = AmazonConverseConfig() + schema = { + "type": "object", + "properties": {"answer": {"type": "string"}}, + } + + with patch.object( + AmazonConverseConfig, + "_supports_native_structured_outputs", + return_value=False, + ): + with caplog.at_level("WARNING"): + result = config._transform_request( + model="bedrock/converse/us.anthropic.claude-3-haiku-20240307-v1:0", + messages=[{"role": "user", "content": "hi"}], + optional_params={ + "maxTokens": 256, + "output_config": { + "format": {"type": "json_schema", "schema": schema}, + }, + }, + litellm_params={}, + headers={}, + ) + + assert "outputConfig" not in result + assert any( + "dropping `output_config.format`" in record.getMessage() + for record in caplog.records + ) + + +def test_output_config_normalized_marker_does_not_leak_into_optional_params(): + """The internal ``_output_config_normalized`` marker set by + ``_handle_reasoning_effort_parameter`` must be consumed during request + preparation so it does not linger on the caller's ``optional_params``.""" + config = AmazonConverseConfig() + + optional_params = config.map_openai_params( + non_default_params={"reasoning_effort": "xhigh"}, + optional_params={}, + model="bedrock/converse/us.anthropic.claude-opus-4-6-v1", + drop_params=False, + ) + assert optional_params.get("_output_config_normalized") is True + + config._transform_request( + model="bedrock/converse/us.anthropic.claude-opus-4-6-v1", + messages=[{"role": "user", "content": "hi"}], + optional_params=optional_params, + litellm_params={}, + headers={}, + ) + + assert "_output_config_normalized" not in optional_params + + +@pytest.mark.parametrize( + "model,expected_effort", + [ + ("bedrock/converse/us.anthropic.claude-opus-4-5-20251101-v1:0", "high"), + ("bedrock/converse/us.anthropic.claude-opus-4-6-v1", "max"), + ("bedrock/converse/us.anthropic.claude-opus-4-7", "xhigh"), + ], +) +def test_output_config_effort_normalized_for_bedrock_converse_opus( + model, expected_effort +): + """Bedrock Converse accepts ``xhigh`` and forwards the provider-safe effort.""" + config = AmazonConverseConfig() + + result = config._transform_request( + model=model, + messages=[{"role": "user", "content": "hi"}], + optional_params={ + "maxTokens": 256, + "thinking": {"type": "adaptive"}, + "output_config": {"effort": "xhigh"}, + }, + litellm_params={}, + headers={}, + ) + + additional = result.get("additionalModelRequestFields", {}) + assert additional.get("output_config") == {"effort": expected_effort} + + @pytest.mark.parametrize( "effort", ["disabled", "invalid", ""], diff --git a/tests/test_litellm/llms/bedrock/messages/invoke_transformations/test_anthropic_claude3_transformation.py b/tests/test_litellm/llms/bedrock/messages/invoke_transformations/test_anthropic_claude3_transformation.py index 2e315a535f..c92a990522 100644 --- a/tests/test_litellm/llms/bedrock/messages/invoke_transformations/test_anthropic_claude3_transformation.py +++ b/tests/test_litellm/llms/bedrock/messages/invoke_transformations/test_anthropic_claude3_transformation.py @@ -767,6 +767,163 @@ def test_bedrock_messages_forwards_output_config_with_output_format(): assert "output_format" not in result +def test_bedrock_messages_converts_output_config_format_to_inline_schema(): + """``output_config.format`` is consumed so Bedrock does not see an unknown nested key.""" + from unittest.mock import patch + + from litellm.types.router import GenericLiteLLMParams + + cfg = AmazonAnthropicClaudeMessagesConfig() + messages = [{"role": "user", "content": [{"type": "text", "text": "Hello"}]}] + schema = { + "type": "object", + "properties": {"answer": {"type": "string"}}, + } + optional_params = { + "max_tokens": 4096, + "output_config": { + "effort": "xhigh", + "format": {"type": "json_schema", "schema": schema}, + }, + } + + with patch( + "litellm.llms.bedrock.messages.invoke_transformations.anthropic_claude3_transformation._supports_factory", + return_value=True, + ): + result = cfg.transform_anthropic_messages_request( + model="anthropic.claude-opus-4-7", + messages=messages, + anthropic_messages_optional_request_params=optional_params, + litellm_params=GenericLiteLLMParams(), + headers={}, + ) + + assert result.get("output_config") == {"effort": "xhigh"} + assert "output_format" not in result + last_content = result["messages"][0]["content"] + assert json.loads(last_content[-1]["text"]) == schema + + +@pytest.mark.parametrize( + "model,expected_effort", + [ + ("anthropic.claude-opus-4-5-20251101-v1:0", "high"), + ("anthropic.claude-opus-4-6-v1", "max"), + ("anthropic.claude-opus-4-7", "xhigh"), + ], +) +def test_bedrock_messages_normalizes_output_config_effort_for_opus( + model, expected_effort +): + """Bedrock /v1/messages accepts ``xhigh`` and forwards the provider-safe effort.""" + from unittest.mock import patch + + from litellm.types.router import GenericLiteLLMParams + + cfg = AmazonAnthropicClaudeMessagesConfig() + + with patch( + "litellm.llms.bedrock.messages.invoke_transformations.anthropic_claude3_transformation._supports_factory", + return_value=True, + ): + result = cfg.transform_anthropic_messages_request( + model=model, + messages=[{"role": "user", "content": [{"type": "text", "text": "Hello"}]}], + anthropic_messages_optional_request_params={ + "max_tokens": 4096, + "output_config": {"effort": "xhigh"}, + }, + litellm_params=GenericLiteLLMParams(), + headers={}, + ) + + assert result.get("output_config") == {"effort": expected_effort} + + +def test_bedrock_messages_does_not_mutate_callers_messages_when_embedding_schema(): + """Inline-schema embedding must not mutate the caller's ``messages`` list, + message dicts, or content list.""" + from unittest.mock import patch + + from litellm.types.router import GenericLiteLLMParams + + cfg = AmazonAnthropicClaudeMessagesConfig() + caller_content = [{"type": "text", "text": "Hello"}] + caller_message = {"role": "user", "content": caller_content} + caller_messages = [caller_message] + schema = {"type": "object", "properties": {"answer": {"type": "string"}}} + optional_params = { + "max_tokens": 4096, + "output_config": { + "effort": "xhigh", + "format": {"type": "json_schema", "schema": schema}, + }, + } + + with patch( + "litellm.llms.bedrock.messages.invoke_transformations.anthropic_claude3_transformation._supports_factory", + return_value=True, + ): + result = cfg.transform_anthropic_messages_request( + model="anthropic.claude-opus-4-7", + messages=caller_messages, + anthropic_messages_optional_request_params=optional_params, + litellm_params=GenericLiteLLMParams(), + headers={}, + ) + + assert caller_messages == [ + {"role": "user", "content": [{"type": "text", "text": "Hello"}]} + ] + assert caller_message == { + "role": "user", + "content": [{"type": "text", "text": "Hello"}], + } + assert caller_content == [{"type": "text", "text": "Hello"}] + last_content = result["messages"][-1]["content"] + assert json.loads(last_content[-1]["text"]) == schema + + +def test_bedrock_messages_does_not_mutate_callers_output_config(): + """`pop_bedrock_invoke_output_config_format` / effort normalization must not + leak into the caller's ``optional_params`` dict.""" + from unittest.mock import patch + + from litellm.types.router import GenericLiteLLMParams + + cfg = AmazonAnthropicClaudeMessagesConfig() + schema = { + "type": "object", + "properties": {"answer": {"type": "string"}}, + } + caller_output_config = { + "effort": "xhigh", + "format": {"type": "json_schema", "schema": schema}, + } + optional_params = { + "max_tokens": 4096, + "output_config": caller_output_config, + } + + with patch( + "litellm.llms.bedrock.messages.invoke_transformations.anthropic_claude3_transformation._supports_factory", + return_value=True, + ): + cfg.transform_anthropic_messages_request( + model="anthropic.claude-opus-4-5-20251101-v1:0", + messages=[{"role": "user", "content": [{"type": "text", "text": "Hello"}]}], + anthropic_messages_optional_request_params=optional_params, + litellm_params=GenericLiteLLMParams(), + headers={}, + ) + + assert caller_output_config == { + "effort": "xhigh", + "format": {"type": "json_schema", "schema": schema}, + } + + def test_bedrock_messages_strips_output_config_with_output_format(): """ When both output_config and output_format are present, output_format @@ -1071,9 +1228,7 @@ def test_bedrock_messages_preserves_compact_context_management_and_adds_beta(): messages = [{"role": "user", "content": [{"type": "text", "text": "Hi"}]}] optional_params = { "max_tokens": 4096, - "context_management": { - "edits": [{"type": "compact_20260112"}] - }, + "context_management": {"edits": [{"type": "compact_20260112"}]}, } result = cfg.transform_anthropic_messages_request( @@ -1084,9 +1239,7 @@ def test_bedrock_messages_preserves_compact_context_management_and_adds_beta(): headers={}, ) - assert result.get("context_management") == { - "edits": [{"type": "compact_20260112"}] - } + assert result.get("context_management") == {"edits": [{"type": "compact_20260112"}]} assert "compact-2026-01-12" in result.get("anthropic_beta", []) assert result["max_tokens"] == 4096 @@ -1118,9 +1271,7 @@ def test_bedrock_messages_filters_unsupported_context_management_edits(): headers={}, ) - assert result.get("context_management") == { - "edits": [{"type": "compact_20260112"}] - } + assert result.get("context_management") == {"edits": [{"type": "compact_20260112"}]} assert "compact-2026-01-12" in result.get("anthropic_beta", []) diff --git a/tests/test_litellm/llms/bedrock/test_bedrock_common_utils.py b/tests/test_litellm/llms/bedrock/test_bedrock_common_utils.py index c39fb427a0..6298eeb25e 100644 --- a/tests/test_litellm/llms/bedrock/test_bedrock_common_utils.py +++ b/tests/test_litellm/llms/bedrock/test_bedrock_common_utils.py @@ -1,9 +1,7 @@ -import json import os import sys import pytest -from fastapi.testclient import TestClient sys.path.insert( 0, os.path.abspath("../../../..") @@ -12,7 +10,6 @@ sys.path.insert( from litellm.llms.bedrock.common_utils import BedrockModelInfo - # --------------------------------------------------------------------------- # # get_bedrock_response_stream_shape lazy-load tests # # --------------------------------------------------------------------------- # @@ -24,8 +21,10 @@ def _reset_bedrock_response_stream_shape_cache(): import litellm.llms.bedrock.common_utils as mod mod.get_bedrock_response_stream_shape.cache_clear() + mod._get_local_model_cost_map.cache_clear() yield mod.get_bedrock_response_stream_shape.cache_clear() + mod._get_local_model_cost_map.cache_clear() def test_bedrock_response_stream_shape_lazy_loads_once(): @@ -222,3 +221,45 @@ def test_context_window_suffix_stripped_for_cost_lookup(): get_bedrock_base_model("anthropic.claude-3-5-sonnet-20241022-v2:0:51k") == "anthropic.claude-3-5-sonnet-20241022-v2:0" ) + + +def test_output_config_effort_normalization_uses_model_info_ceiling(monkeypatch): + import litellm.llms.bedrock.common_utils as mod + + calls = [] + + def fake_get_model_info(model, custom_llm_provider=None): + calls.append((model, custom_llm_provider)) + return {"bedrock_output_config_effort_ceiling": "max"} + + monkeypatch.setattr(mod, "_get_model_info", fake_get_model_info) + output_config = {"effort": "xhigh"} + + mod.normalize_bedrock_opus_output_config_effort( + model="custom-bedrock-alias-without-opus-pattern", + output_config=output_config, + ) + + assert output_config == {"effort": "max"} + assert calls == [("custom-bedrock-alias-without-opus-pattern", "bedrock")] + + +@pytest.mark.parametrize( + "model,expected_ceiling", + [ + ("anthropic.claude-opus-4-5-20251101-v1:0", "high"), + ("anthropic.claude-opus-4-6-v1", "max"), + ("anthropic.claude-opus-4-7", "xhigh"), + ("us.anthropic.claude-opus-4-5-20251101-v1:0", "high"), + ("us.anthropic.claude-opus-4-6-v1", "max"), + ("us.anthropic.claude-opus-4-7", "xhigh"), + ], +) +def test_bundled_bedrock_opus_model_info_declares_output_config_effort_ceiling( + model, expected_ceiling +): + from litellm.litellm_core_utils.get_model_cost_map import GetModelCostMap + + model_info = GetModelCostMap.load_local_model_cost_map()[model] + + assert model_info["bedrock_output_config_effort_ceiling"] == expected_ceiling diff --git a/tests/test_litellm/test_utils.py b/tests/test_litellm/test_utils.py index e646c75eda..eaa875531e 100644 --- a/tests/test_litellm/test_utils.py +++ b/tests/test_litellm/test_utils.py @@ -859,7 +859,11 @@ def test_aaamodel_prices_and_context_window_json_is_valid(): "supports_adaptive_thinking": {"type": "boolean"}, "supports_service_tier": {"type": "boolean"}, "supports_preset": {"type": "boolean"}, - "supports_output_config": {"type": "boolean"}, + "supports_output_config": {"type": "boolean"}, + "bedrock_output_config_effort_ceiling": { + "type": "string", + "enum": ["low", "medium", "high", "max", "xhigh"], + }, "tool_use_system_prompt_tokens": {"type": "number"}, "tpm": {"type": "number"}, "provider_specific_entry": {"type": "object"},