The OpenAI SDK raw_response.parse() can return a plain str instead
of a Pydantic model when Azure returns a non-JSON content type (e.g.,
HTML error page, proxy error). Calling .model_dump() on the str then
raises AttributeError.
Adds isinstance(response, str) checks before all 4 model_dump() call
sites in the Azure chat completion and embedding paths.
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Parse VIDEO modality in promptTokensDetails → prompt_tokens_details.video_tokens
- Parse VIDEO modality in candidatesTokensDetails → completion_tokens_details.video_tokens
- Parse VIDEO modality in cacheTokensDetails and subtract from prompt video tokens
- Add video_tokens field to PromptTokensDetailsWrapper and CompletionTokensDetailsWrapper
- Fix implicit caching text-token fallback to not fire when cacheTokensDetails is present
- Add 4 unit tests covering: prompt video tokens, response video tokens,
auto-calculated text fallback with video, and explicit video cache subtraction
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Covers:
- Region + modelId correctly extracted for ap-northeast-1, us-east-1, us-west-2
- No region in path leaves modelId and optional_params unchanged
- Cross-region inference prefixes (us., eu., ap.) are not treated as region segments
- Explicitly set aws_region_name is not overridden by region in model path
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
When a user passes model="bedrock/ap-northeast-1/moonshotai.kimi-k2.5", get_llm_provider
strips the "bedrock/" prefix and passes "ap-northeast-1/moonshotai.kimi-k2.5" to the
converse handler. Two bugs occurred:
1. modelId was encoded as "ap-northeast-1%2Fmoonshotai.kimi-k2.5" (region included),
which AWS rejects as "not a valid model identifier"
2. The region ap-northeast-1 was never extracted, so the request went to the wrong
default region instead
Fix: after stripping routing prefixes in converse_handler.py completion(), check if the
remaining path starts with a known AWS region and strip it from modelId, injecting it
into optional_params so _get_aws_region_name picks it up.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Extract the repeated usage-combining block from both
_completion_streaming_iterator and _acompletion_streaming_iterator into a
shared static helper method _combine_fallback_usage. This brings both
functions under the PLR0915 50-statement limit, removing the noqa
suppressions.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add `# noqa: PLR0915` suppression to match the async twin
`_acompletion_streaming_iterator` which already carries the same
suppression. The function's complexity is inherent (nested class,
generator with fallback logic, cleanup code).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The observatory test workflow failed because the "Verify tunnel
connectivity" step used a single curl with no retries. Cloudflare quick
tunnels need time for DNS propagation, and the first lookup can return
NXDOMAIN (curl exit 6). Replace with a retry loop (10 attempts, 5s
apart) matching the pattern already used in the health check step.
Also add `# noqa: PLR0915` to `_completion_streaming_iterator` in
router.py, matching the suppression already on its async twin.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
actions/checkout treats short commit hashes as branch names, causing
fetch failures. The checkout only needs the config file from the
repo, so use the default branch instead of a specific ref.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The secrets context is not available in step-level if: conditions,
causing the workflow file to fail validation. Move the conditional
check into the shell script instead.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Pass AZURE_API_KEY, AZURE_API_BASE, OBSERVATORY_URL,
OBSERVATORY_API_KEY, and REQUEST_ID through step-level env
blocks so they are never interpolated directly into shell scripts.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Validate inputs.tag matches vX.Y.Z format to prevent script
injection via workflow_dispatch
- Pass tag via env var instead of direct interpolation in shell
- Add cleanup step to kill cloudflared and remove docker container
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add timeout-minutes: 30 to prevent runaway jobs
- Build /run-test payload with jq --arg to safely escape
TUNNEL_URL and LITELLM_MASTER_KEY values
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Fail early if request_id is missing or null from the /run-test
response instead of polling /run-status/null for 15 minutes.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Avoids shell quoting issues with single quotes in JSON and
multi-line output truncation when using GITHUB_OUTPUT.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add permissions block (contents: read) per GitHub security scan
- Poll /run-status/{request_id} instead of global /queue-status
to avoid race conditions with concurrent test runs
- Add result verification step that fails the workflow if tests
did not pass or the run errored
- Fix auth header to use X-LiteLLM-Observatory-API-Key
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- New reusable workflow that spins up a LiteLLM container from the
release image, exposes it via cloudflared tunnel, and triggers
test runs on the Railway-hosted observatory
- Integrates into ghcr_deploy.yml for RC and stable releases
- Can also be triggered manually via workflow_dispatch
- Add placeholder litellm_config.yaml for observatory test models
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The transformation.py file was using FEATHERLESS_API_KEY (missing _AI_)
while the rest of the codebase (get_llm_provider_logic.py, utils.py)
correctly uses FEATHERLESS_AI_API_KEY. This caused 401 auth errors when
the user set FEATHERLESS_AI_API_KEY as documented.
Now checks FEATHERLESS_AI_API_KEY first (canonical name) with fallback
to FEATHERLESS_API_KEY (legacy compatibility). Same fix applied to
FEATHERLESS_AI_API_BASE.
Refs: #22490
Add detailed UI walkthrough for Project Management feature including:
- Beta notice with link to API documentation
- Overview of projects and organizational hierarchy
- Prerequisites and setup instructions
- Separate section for enabling projects in UI settings
- Step-by-step guide for creating and managing projects
- Use cases for key organization within teams
- Next steps and related documentation links
- Proper sidebar navigation integration
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
A new QueryClient() was instantiated inside 5 component render functions
and at module-level in 3 more pages, with no shared QueryClientProvider
in either layout. This caused isolated, ephemeral caches with no
cross-page sharing and cache destruction on every re-render.
Moves QueryClient to a single module-level constant in AntdGlobalProvider
(the existing root-level "use client" wrapper in app/layout.tsx) and
removes the per-page QueryClient instantiations and QueryClientProvider
wrappers from all 8 affected pages.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Enables guardrail processing for OCR requests and responses. Adds OCR handler under litellm/llms/mistral/ocr/guardrail_translation/ to process document URLs on input and extracted page markdown on output. Includes route-to-call-type mappings for /ocr and /v1/ocr endpoints. Adds 14 unit tests and 4 e2e tests verifying handler discovery, input/output processing, and integration with UnifiedLLMGuardrails.
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Moves is_accepted=True from GET /onboarding/get_token to POST /onboarding/claim_token,
so the flag accurately reflects that a password has been set. Both endpoints now reject
already-used links, with get_token rejecting before any user data is returned.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* feat(ui): add missing provider logos and map all backend providers to UI
- Downloaded 26 SVG logos from lobehub/lobe-icons for providers that were
missing visual branding (AI21, Baseten, Cloudflare, GitHub, Huggingface,
Hyperbolic, Lambda, LM Studio, Meta Llama, Moonshot, Nebius, Novita,
Nvidia NIM, Replicate, Recraft, Topaz, V0, Vercel, Watsonx/IBM,
Xinference, Friendli, Morph, Cometapi, Featherless, Langfuse, GitHub Copilot)
- Extended Providers enum from 47 to 107 entries to cover all backend
providers from provider_create_fields.json
- Extended provider_map to map all new enum keys to litellm_provider values
- Extended providerLogoMap to assign logos to all providers where available,
reusing parent logos for variants (e.g. Anthropic Text -> anthropic.svg)
- Fixed SVG currentColor issue: replaced fill='currentColor' with explicit
colors since CSS inheritance doesn't work in <img> elements
- Updated test reference from Providers.Watsonx to Providers.WATSONX
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* docs(agents): add UI dashboard dev notes to Cursor Cloud instructions
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* refactor(ui): remove non-LLM providers from Add Model dropdown
Remove Custom, Custom OpenAI, GitHub, Humanloop, Langfuse, Litellm Proxy,
and Milvus from the Providers enum, provider_map, and providerLogoMap.
These are not LLM API providers (they are internal tools, vector stores,
or observability platforms) and should not appear in the Add Model form.
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
The `gemini/gemini-2.5-flash-image` entry had `litellm_provider` set to
`vertex_ai-language-models` instead of `gemini`. This causes a provider
mismatch in `_check_provider_match()` when the model is used via the
Gemini API provider (`custom_llm_provider="gemini"`), resulting in a
noisy error log on every request:
"This model isn't mapped yet. model=gemini/gemini-2.5-flash-image,
custom_llm_provider=gemini"
The `vertex_ai/gemini-2.5-flash-image` entry already exists with the
correct `vertex_ai-language-models` provider, and the sibling
`gemini/gemini-2.5-flash-image-preview` entry correctly uses `gemini`.
Some OpenAI-compatible providers return null for top_logprobs when
logprobs=true but top_logprobs is unset or 0. The OpenAI spec requires
top_logprobs to be Array<TopLogprob> (never null), so this triggers
Pydantic validation errors while parsing responses.
Add a Pydantic v2 field_validator on ChatCompletionTokenLogprob that
normalizes None -> [] before type validation. This preserves the typed
List[TopLogprob] contract for downstream consumers while remaining
narrowly scoped to null only (other invalid types are still rejected).
Fixes#21932
* fix: add sync streaming mid-stream fallback + fix 429 for all streaming paths
Some LiteLLM providers (Vertex AI, Bedrock, Predibase, Codestral) use a
deferred HTTP pattern where the streaming HTTP request is made lazily on
the first iteration, not during completion()/acompletion(). This means
errors surface during __next__/__anext__, outside the Router's
retry/fallback machinery.
Two gaps existed:
1. __anext__ had a blanket 4xx filter (PR #18698) that blocked 429 from
MidStreamFallbackError — fixed here by exempting 429.
2. __next__ had NO MidStreamFallbackError support at all, and the Router
had no sync streaming fallback wrapper — both added here.
Changes:
- streaming_handler.py: Extract shared _handle_stream_fallback_error()
used by both __next__ and __anext__. Maps exceptions, filters
non-retriable 4xx (excluding 429), wraps everything else in
MidStreamFallbackError.
- router.py: Add _completion_streaming_iterator() (sync mirror of
_acompletion_streaming_iterator). Modify _completion() to wrap
streaming responses. Add is_pre_first_chunk check to both async
and sync iterators to skip continuation prompt on pre-call errors.
Fixes#22296
Relates to #20870, #8648, #6532
* fix: no-op assertion in sync streaming fallback test
The assertion `... is None or True` always evaluated to True,
meaning it never actually verified anything. Replace with a
proper check that messages match the original (no continuation
prompt on pre-first-chunk errors).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* fix(anthropic): populate output_config when reasoning_effort is used on Claude 4.6
When reasoning_effort is passed for Claude 4.6 models, _map_reasoning_effort
returns {type: 'adaptive'} but the effort level is silently dropped. Per
the Anthropic docs, effort on 4.6 models is controlled via output_config,
not thinking budget_tokens.
Map reasoning_effort to output_config.effort for 4.6 models so the effort
guidance is sent to the API.
Fixes#22212
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* test: add coverage for "max" effort level in Claude 4.6 reasoning test
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
