Three follow-ups to the OAuth-discovery SSRF guard:
1. Greptile P1 (redirect bypass): the validated origin could return a
3xx whose ``Location`` points at an internal address, and httpx
would follow without re-checking the new target. Pass
``follow_redirects=False`` to both gated httpx GETs. Spec-compliant
OAuth/OIDC metadata endpoints serve the JSON directly, so this
doesn't affect legitimate providers.
2. Greptile P2 (empty getaddrinfo): POSIX doesn't strictly forbid an
empty success-list from ``getaddrinfo``. Add an explicit
``if not infos: return False`` so the guard fails closed instead of
falling through to ``return True``.
3. Mypy: ``info[4][0]`` is typed ``str | int``; narrow at the
boundary with an ``isinstance`` check (fail-closed if non-str).
Adds two regression tests verifying ``follow_redirects=False`` is
passed at both gated fetch sites, and one verifying the empty-list
case rejects the URL.
The OAuth discovery code in mcp_server_manager followed two
attacker-influenceable URLs without validation: the
``resource_metadata`` URL parsed out of a ``WWW-Authenticate``
challenge, and the ``authorization_servers[0]`` field of the
PRM JSON returned by the resource server. A malicious MCP server
could point those at a cloud-instance-metadata service, an internal
admin panel, or a loopback debug endpoint and the proxy would issue
a blind GET on its behalf.
Add ``_is_safe_metadata_url(url, server_url)`` and gate both follow-
up fetch sites on it. A URL is allowed when:
- it shares scheme + host + port with ``server_url`` (well-known
endpoints constructed from the admin's URL, and PRM published at
the resource server itself per RFC 9728 §3.3), or
- it resolves to publicly-routable IPs only (covers federated
authorization servers — Azure Entra, Google, Okta, GitHub —
hosted cross-origin from the resource server).
URLs that resolve to private / loopback / link-local / cloud-metadata
addresses, or that don't resolve at all, are rejected. ``http`` and
``https`` are the only schemes accepted. The IP block list is
provided by the existing ``_is_blocked_ip`` helper from
``litellm_core_utils.url_utils`` so the policy stays consistent with
the rest of the proxy.
The guard does not protect against active DNS rebinding between
this resolution and the subsequent httpx GET — the same-authority
pin remains the primary mitigation; the IP check is defence in
depth. The surface only triggers on config load / add-server, not
per request, so the synchronous ``getaddrinfo`` is acceptable.
Threads ``server_url`` through ``_fetch_oauth_metadata_from_resource``,
``_fetch_authorization_server_metadata``, and
``_fetch_single_authorization_server_metadata``. Existing tests for
those helpers updated for the new signature; new
``TestOAuthDiscoverySSRFGuard`` covers same-authority allow,
private-IP rejection across IPv4 and IPv6, multi-A-record dual-
stack rejection, unresolvable hosts, non-http schemes, and
end-to-end "no network call when guard denies".
Greptile P1: this PR encrypts LiteLLM_MCPUserCredentials rows under the
salt key, but the /key/regenerate rotation endpoint had no
corresponding step for that table. Rotating the master key would
leave every BYOK and OAuth2 user credential permanently unreadable.
Adds rotate_mcp_user_credentials_master_key, mirroring the existing
rotate_mcp_server_credentials_master_key pattern: read each row with
the current key (via _decode_user_credential, which also handles
unmigrated legacy plaintext rows), re-encrypt under the new master
key, write back. One bad row is logged and skipped instead of
aborting the whole rotation.
Wired into key_management_endpoints.py as step 4b, alongside the
existing server-credentials rotation, with the same try/except shape
so a transient DB error on this table doesn't kill the whole
regenerate-key flow.
Tests cover: round-trip through rotation under a new key, automatic
re-encryption of legacy plaintext rows (rotation also acts as a
migration trigger), and a corrupt row not aborting the rotation.
Three minor fixes from Greptile review:
1. _decode_user_credential now also catches TypeError so a null
credential_b64 value returns None instead of propagating, matching
the documented "returns None when neither path yields a valid
string" contract.
2. The OAuth2 BYOK guard error no longer claims the existing row is a
BYOK credential — after a salt-key rotation, an OAuth2 row can fail
to decrypt and reach the same guard. Reword to "could not be
verified as an OAuth2 token", which is accurate for both cases.
3. Drop the no-op sys.path.insert in the new test file (other tests
in the directory don't need it; pytest picks up the package via
the installed editable wheel).
Adds a regression test for the None-input case.
LiteLLM_MCPUserCredentials.credential_b64 stored both BYOK API keys and
OAuth2 access tokens as plain urlsafe-base64 of the raw value. Any DB
read could recover the upstream-provider key.
Run all writes through encrypt_value_helper (nacl SecretBox, the same
helper used for the server-level credentials column) and read back via
a small dual-path helper that tries decryption first, then falls back to
plain base64 so existing rows keep working until they get rewritten.
Folds the three near-identical "decode -> json.loads -> check type ==
oauth2" sites into _decode_oauth_payload, which simplifies the BYOK
guard inside store_user_oauth_credential.
- server.py: drop the redundant server_id append in
_get_filtered_mcp_servers_from_mcp_server_names. iter_known_server_prefixes
already yields server_id unconditionally, so the manual append (and its
misleading comment) was a no-op duplicate.
- utils.py: rewrite the SHORT_MCP_TOOL_PREFIX docstring to accurately
describe the collision behaviour. The previous wording said collisions
were 'cosmetic only', but a natural-hash collision IS a routing-correctness
issue, which is precisely why we already added _assign_unique_short_prefix
to rehash deterministically. The new comment cross-references that path.
- utils.py: restrict the first character of the short prefix to [A-Za-z]
via a 52-char alphabet for position 0 only. The remaining two positions
still use the full base62 alphabet. This keeps prefixes valid identifiers
on every backend and gives 52*62*62 = 199_888 distinct prefixes (still
comfortably more than any realistic deployment).
- tests: add coverage proving the first character of the prefix is always
alphabetic across many server_ids and rehash attempts.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
Two MCP servers can natural-hash to the same three-character base62
prefix. With 62**3 = 238_328 slots the birthday bound is ~488 servers
for 50% collision probability, so a single proxy hosting more than
~100 MCP servers has a non-trivial chance of seeing a collision in
practice — and a collision means tool names from two different servers
share a routing key, causing silent mis-routing.
Mitigation:
- compute_short_server_prefix(server_id, attempt=N) folds an attempt
counter into the SHA-256 seed, so rehashes are deterministic and
produce a fresh three-char prefix space per attempt.
- New MCPServer.short_prefix field caches the resolved (post-dedup)
prefix on the model so it stays stable across the process lifetime.
- MCPServerManager._assign_unique_short_prefix walks attempts 0..N
until it finds a prefix not already used by another server in the
combined registry. Logs an INFO line when a rehash happens so
operators have a breadcrumb if it ever does.
- Wired into every registration path: load_servers_from_config,
add_server, update_server, reload_servers_from_database. The
database reload path also carries the previously-resolved prefix
forward so reloads don't churn it.
- get_server_prefix prefers the cached short_prefix when set, so the
resolved value (not the raw natural hash) is used everywhere.
- iter_known_server_prefixes yields the cached short_prefix too, so
reverse-lookup tolerance covers the rehashed form.
No-op when LITELLM_USE_SHORT_MCP_TOOL_PREFIX is disabled — the field
stays None and behaviour is unchanged.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
Adds LITELLM_USE_SHORT_MCP_TOOL_PREFIX. When enabled, tool / prompt /
resource / resource-template names emitted from MCP servers are prefixed
with a deterministic three-character base62 ID derived from the server's
server_id (SHA-256 → base62) instead of the (potentially long)
alias / server_name. This keeps namespaced tool names well under the
60-character upper bound enforced by some model APIs while still letting
us distinguish MCP-routed tools from local tools.
Behavioural notes:
- Default off — when the env var is unset, the long-prefix behaviour
is unchanged. The plan is to flip the default in a future release
and remove the gate after a deprecation window.
- Prefix derivation is deterministic, so it is stable across processes,
workers and restarts without any persistence layer.
- Reverse-lookup is tolerant: _create_prefixed_tools registers every
known prefix form (alias / server_name / server_id / short ID) in
the routing map and _get_mcp_server_from_tool_name resolves any of
them. Old clients holding cached long-prefixed names continue to
route correctly even after the flag is enabled.
- _get_allowed_mcp_servers_from_mcp_server_names accepts the short
prefix in /mcp/{server_name}-style URLs.
- The OpenAPI tool-listing path now filters by the active server
prefix instead of server.name so spec-backed servers benefit too.
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
Greptile review feedback (P2): the two negative `.well-known`-substring
tests fell through to `_target_servers_use_oauth2`, which queries
`global_mcp_server_manager.get_mcp_server_by_name`. Without an explicit
mock the tests passed only because the real registry happens to be empty
in the test process. Mock the manager to return None so the assertion
exercises the fail-closed path explicitly.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Greptile flagged a regression introduced in the previous commit's merged
exception handler: ``ProxyException.__init__`` normalizes ``code`` via
``str(code)``, so a ``code=None`` (valid per the type signature) becomes
the string ``"None"``. Coercing that with ``int(...)`` raises
``ValueError``, which propagates uncaught and rewrites the auth error as
an unhandled 500 — degrading security posture compared to the pre-merge
``str(e.code) in ("401", "403")`` shape.
Compare against both int and str forms of the auth-error codes instead
of coercing. Adds a regression test for the ``code=None`` case.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two related issues in `MCPRequestHandler.process_mcp_request`:
1. Public-route detection used `".well-known" in str(request.url)`, a
substring match against the full URL. Attackers could smuggle the
marker via the query string, hostname, or a deeper path segment to
bypass authentication on any MCP route. Replaced with an exact path
prefix on `request.url.path` (`startswith("/.well-known/")`).
2. The OAuth2 passthrough fallback (added in #20602 to support
`auth_type=oauth2` upstream MCP servers like Atlassian) caught any
401/403 from `user_api_key_auth` and replaced the result with an
anonymous `UserAPIKeyAuth()`. That fallback fired regardless of the
target server's configured `auth_type`, so an attacker presenting a
garbage `Authorization` header could exchange a failed LiteLLM auth
for an anonymous session against any server. The fallback now runs
only when EVERY MCP server the request targets is operator-configured
for `auth_type=oauth2`. For any non-oauth2 server (api_key,
bearer_token, basic, etc.), the auth error propagates as before.
Target resolution prefers the `x-mcp-servers` header when present
(including the explicitly-empty case, which fails closed) and otherwise
parses the standard `/mcp/{server_name}` and `/{server_name}/mcp`
transport URL patterns. Routes that don't match either form fail closed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`_check_byok_credential` previously returned silently when `prisma_client`
was None, bypassing BYOK ownership validation during database-outage
windows. Any proxy-authenticated user could invoke BYOK-protected MCP
tools without a stored credential during the outage window.
Now raises HTTP 503 with a structured error so the flow fails closed.
Regression test asserts 503 is raised when `prisma_client` is None.
Reported by @brodmart in GHSA-6762-2m23-5mxp.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
PR feedback (greptile P1 / veria high): with the previous change, a team
storing mcp_tool_permissions={"my-alias": ["read_file"]} would pass the
server-access check (because the alias expanded to a concrete id in the
allowed-servers list) but the per-server tool lookup still did
dict.get(server_id) against the raw name-keyed dict — missing, returning
None, which callers treat as "no restrictions" → all tools allowed instead
of only the declared ones.
Add MCPServerManager.expand_tool_permissions() that rewrites the dict so
every key is a concrete server_id where possible (tool lists from keys
pointing at the same server are unioned). Unresolved keys pass through
unchanged so stale id-keyed restrictions still apply when the same string
is used for lookup. Wire the helper into the four dict-lookup sites:
get_allowed_tools_for_server (key + team paths), the agent tool lookup,
and the rest_endpoints.py tool filter.
Also switch expand_permission_list to pass through unresolved entries
(rather than dropping them) so existing test fixtures that use bare string
placeholders continue to work. The downstream access check denies unknown
entries when compared to the concrete request server_id, so security
posture is unchanged.
Sanitize the debug log to use %r formatting so an admin-controlled
identifier with newlines can't forge log entries (CodeQL log-injection
warning).
team.object_permission.mcp_servers (and the per-key equivalent) previously
only accepted server_id strings. For config-loaded MCP servers, the id is
derived from a hash that includes the server URL, so the same logical
server in two regions ends up with two different ids in a shared database.
Permission lists had to enumerate every region's id.
Add a single MCPServerManager.expand_permission_list() helper that resolves
each entry against the current region's config + DB registry union: entries
that match a server_id pass through, entries that match an alias/server_name/
name expand to every matching id, and unresolved entries drop with a debug
log so stale or typo entries are diagnosable. Wire it into the four
_get_allowed_mcp_servers_for_* helpers so direct server entries and
mcp_tool_permissions dict keys are both expanded before the intersection.
Access-check outcomes are unchanged for existing id-based permissions;
name-based entries now resolve instead of being silently denied.
Address codex review P1 + P2 findings:
- BYOK /token now accepts OAuth 2.1 clients that omit redirect_uri
(draft-15 §4.1.3 dropped the requirement). When the client does
submit a value, equality is still enforced vs the /authorize record.
PKCE + client_id binding cover the security role redirect_uri
played under RFC 6749.
- _user_id_from_session_cookie requires the ``exp`` claim on the UI
session JWT (PyJWT options={"require": ["exp"]}) so leaked cookies
have a bounded lifetime.
- validate_loopback_redirect_uri rejects URIs with a fragment
(RFC 6749 §3.1.2) and catches malformed-URI ValueError so
unparseable input surfaces as 400 invalid_request instead of 500.
Address codex review P0 + P1 findings on the discoverable OAuth proxy:
- /callback now re-validates that the decoded base_url is loopback before
302-redirecting to it. State is encrypted but pre-existing states minted
before the /authorize validation was added have no expiry and remain
valid; validating at the sink closes the open-redirect + code-theft
primitive for those stale states too. (VERIA-57 root cause B, P0.)
- /token responses now set Cache-Control: no-store + Pragma: no-cache
per RFC 6749 §5.1 (P1).
- Move TOKEN_NO_CACHE_HEADERS constant from byok_oauth_endpoints into
the shared oauth_utils module so both endpoints use the same value.
Extract the BYOK loopback redirect_uri check into a shared
oauth_utils.validate_loopback_redirect_uri helper. Call it in
discoverable_endpoints.authorize_with_server before the client-supplied
redirect_uri is encrypted into the OAuth state.
Without this check, a non-loopback redirect_uri was encoded into the
state parameter and decoded on /callback to 302 the user back to the
attacker's URL with the authorization code attached — an open-redirect
+ code-theft primitive (VERIA-57 root cause B). The /callback handler
is already safe because state is HMAC-signed via encrypt_value_helper,
so validating at /authorize before encoding is sufficient.
Also updates existing tests to use loopback client redirect_uris and
adds regression tests for non-loopback rejection, IPv4 127.0.0.0/8
range acceptance, and full-form IPv6 loopback acceptance.
* fix(mcp_semantic_tool_filter): match canonical tools that arrive with
a client-side namespace prefix.
`SemanticMCPToolFilter._get_tools_by_names` matched by exact equality
between the canonical name stored in the router
(`<server><MCP_TOOL_PREFIX_SEPARATOR><tool>`) and the name in the
incoming `tools[]` list. MCP clients such as opencode wrap every tool
name with their own additive alias prefix
(`<client_alias>_<canonical>`), so the two never matched, the filter
dropped every tool to zero, and the proxy forwarded `tools: []` with
`tool_choice: auto` — which strict upstream providers reject with a 400.
The fix adds anchored suffix matching with a separator check: the
canonical must form the complete tail of the incoming name and be
preceded by `_` or `-`. Exact matches still win over suffix matches,
incoming tools are returned at most once, and the original tool object
is passed through unchanged so the client-facing name survives for
tool-call round-trips.
Seven unit tests in a new TestGetToolsByNames class cover exact
match, underscore- and dash-prefixed variants, non-separator-anchored
suffixes (which must not match), exact-wins-over-prefixed precedence,
deduplication when two canonicals suffix-match the same incoming tool,
and ordering-follows-router-output.
Fixes#26078
* review: strengthen the suffix-fallback tie-breaker and the
deduplication regression test (Greptile comments on #26117)
- test_same_tool_not_returned_twice now passes two distinct canonicals
("read_file" and "file") that both suffix-match the same incoming
tool, rather than the same canonical twice, so the assertion
actually exercises the used_ids dedup path instead of the
duplicate-input-list path.
- The suffix fallback in _get_tools_by_names now prefers the shortest
incoming name that still qualifies under the separator-anchored
match. In the one-prefix-per-client opencode scenario this is a
no-op, but in multi-namespace configurations the shortest qualifying
name is the least-wrapped one and is the most defensible deterministic
choice, replacing the dict-insertion-order fallback.
- Adds test_suffix_fallback_prefers_shortest_candidate covering the
new tie-breaker directly.
Still 15 tests passing locally (was 14).
* review(#26117): gate suffix-matching on canonical containing MCP_TOOL_PREFIX_SEPARATOR
@krrish-berri-2 flagged a possible collision in the suffix fallback:
a local user function whose name happens to end in a bare canonical
substring (e.g. my_firecrawl_scrape vs canonical firecrawl_scrape)
would be spuriously selected.
Server-registered MCP tools are always emitted as
<server_name><MCP_TOOL_PREFIX_SEPARATOR><tool_name> via
add_server_prefix_to_name, so a canonical without the separator is
not a namespaced MCP tool and does not warrant suffix matching.
Added that guard to _name_matches_canonical with a regression test
(test_does_not_collide_with_local_function_on_unprefixed_canonical)
that reproduces the collision before the fix and is pinned after.
Pre-existing TestGetToolsByNames fixtures that relied on bare
canonicals (get_weather, search, read_file, write/delete/read) were
switched to realistic server-prefixed ones so they continue to
exercise the suffix-fallback path under the new guard. The opencode
scenario (client prefix on already-server-prefixed canonical) is
unchanged.
---------
Co-authored-by: sakenuGOD <sakenuGOD@users.noreply.github.com>
Co-authored-by: Krrish Dholakia <krrish+github@berri.ai>
Address two Greptile P2 findings on the BYOK OAuth endpoints:
- GET /v1/mcp/oauth/authorize now runs _validate_redirect_uri up front
so a non-loopback redirect_uri is rejected before the HTML form is
rendered. Previously the user typed an API key, submitted, and got a
400 with no form state.
- POST /v1/mcp/oauth/token moves the master_key guard ahead of the
code-consumption and credential-store steps. Without this, a proxy
with master_key unset would burn the code and persist the credential
but return an error — leaving the user with no way to retrieve a
session token without restarting the whole flow.
- POST /v1/mcp/oauth/authorize now requires an authenticated UI session
cookie. The authenticated user_id — not the OAuth client_id form
field — is stamped onto the authorization code record (RFC 6749 §2.2:
client_id identifies the client, not the user).
- redirect_uri is restricted to loopback per RFC 8252 §7.3 (localhost
plus any ipaddress.is_loopback IP, covering 127.0.0.0/8 and IPv6
loopback forms).
- POST /v1/mcp/oauth/token enforces exact-match of the redirect_uri and
client_id submitted at /authorize (RFC 6749 §4.1.3).
- /token error responses use the RFC 6749 §5.2 format ({"error":
"<code>"}), and all /token responses set Cache-Control: no-store +
Pragma: no-cache (RFC 6749 §5.1).
Per-user OAuth MCP requests now only skip pre-emptive 401 when a stored token is available, preserving token-reuse behavior while restoring fast PKCE kickoff for first-time or missing-token users.
Extend existing test modules with coverage for the instructions merge
logic, upstream cache, ContextVar-based injection, and client-side
capture — following each file's established patterns.
Made-with: Cursor
- Defense-in-depth: warn instead of hard-fail for legacy servers
- Move os import to module level in _types.py
- Document args residual risk in allowlist comment
- Add UpdateMCPServerRequest allowlist test
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add command allowlist for MCP stdio transport to prevent RCE via
/mcp-rest/test/* endpoints. Restrict test endpoints to PROXY_ADMIN
role. Fix docker/README.md MASTER_KEY -> LITELLM_MASTER_KEY.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Fixes#25081.
is_tool_name_prefixed() checked for the presence of MCP_TOOL_PREFIX_SEPARATOR
(default '-') anywhere in the tool name. Any non-MCP tool whose name
contains a hyphen (e.g. 'text-to-speech', 'code-review') was silently
misclassified as an MCP-prefixed tool. When the semantic tool filter is
enabled, these tools would be routed through semantic matching and
potentially dropped.
Fix: accept an optional known_server_prefixes set. When supplied, the
function extracts the candidate prefix (text before the first separator)
and checks it against the normalised set of registered server prefixes.
Only a genuine match returns True. Without the set, legacy behaviour is
preserved for backward compatibility.
Updated _get_mcp_server_from_tool_name() to build the prefix set from
the live registry and pass it through.
9 new tests.
Co-authored-by: d 🔹 <258577966+voidborne-d@users.noreply.github.com>
* added support for metadata (#24261)
* added support for metadata
* fix: PR review - meta truthiness, BlobResourceContents mimeType, add Blob+empty meta tests
Made-with: Cursor
* pyproject to .25
* feat(teams): resolve access group models/MCPs/agents in team endpoints
Add access_group_models, access_group_mcp_server_ids, and
access_group_agent_ids to /team/info and /v2/team/list responses.
These fields contain resources inherited from access groups, kept
separate from direct assignments so the UI can distinguish the source.
Backend: _resolve_access_group_resources() helper resolves access
group resources via existing _get_*_from_access_groups() functions.
UI: Teams table and detail view show direct models as blue badges
and access-group-sourced models as green badges.
* perf(teams): single-pass access group resolution + asyncio.gather in list endpoint
- Fetch each access group object once and extract all 3 resource fields
in a single pass instead of 3 separate calls (3N → N lookups)
- Use asyncio.gather to resolve access groups across teams concurrently
in list_team_v2 instead of sequential awaits
- Add 5 unit tests for _resolve_access_group_resources
* docs: add default_team_params to config reference and update examples
- Add default_team_params to litellm_settings reference table in
config_settings.md with all sub-fields documented
- Update self_serve.md and msft_sso.md examples to include
team_member_permissions, tpm_limit, and rpm_limit
- Fix misleading comment that implied default_team_params only applies
to SSO auto-created teams — it applies to all /team/new calls
* docs: clarify that models sub-field only applies to SSO auto-created teams
* fix: lazy import get_access_object to break cyclic import + short-circuit all-proxy-models display
- Remove get_access_object from module-level import in team_endpoints.py
and use a lazy _get_access_object wrapper to avoid cyclic dependency
- Add _prisma_client is None early-exit guard in _resolve_access_group_resources
- Short-circuit UI to show "All Proxy Models" when team.models is empty
or contains "all-proxy-models", skipping access group model resolution
* add: making organizations a select instead of read only badges
* fix(ui): only send organization_id when changed and use raw initial value
* fix(ui): add paginated team search to usage page filter
Replace the static team dropdown on the usage page with a new
TeamMultiSelect component that uses the paginated v2/team/list
endpoint with debounced server-side search and infinite scroll.
* fix(ui): fix imports and update placeholder for team multi select
* fix(ui): wire team_id filter to key alias dropdown on Virtual Keys tab
The Key Alias dropdown on the Virtual Keys page was showing aliases from
all teams regardless of which team was selected. The team_id was never
passed through the frontend chain to the backend /key/aliases endpoint.
- Backend: add optional team_id query param to /key/aliases endpoint
- networking.tsx: add team_id param to keyAliasesCall
- useKeyAliases: accept and forward team_id to API call and query key
- filter.tsx: pass allFilters context to custom filter components
- PaginatedKeyAliasSelect: read Team ID from allFilters and pass to hook
* fix(tests): correct mock targets in TestResolveAccessGroupResources
Three tests were patching the non-existent `get_access_object` instead
of `_get_access_object` (the lazy-import wrapper), causing AttributeError.
Also added missing `prisma_client` mock so tests get past the early-exit
guard and actually exercise the resolution logic.
* fix: use direct attribute access with or [] fallback in _resolve_access_group_resources
Replace getattr(ag, "field", []) with ag.field or [] for cleaner
access and safe handling if a field is None.
* fix(ui): remove model source legend from team detail view
The blue/green color distinction is self-explanatory; the legend added
visual clutter without providing enough value.
* fix(ui): add missing access_group fields to TeamData.team_info type
The TeamData interface was missing access_group_models,
access_group_mcp_server_ids, and access_group_agent_ids fields,
causing a TypeScript build failure.
* perf(teams): batch-fetch access groups in single DB query
Replace per-ID _resolve_access_group_resources loop with a single
find_many call that deduplicates IDs across all teams. Removes the
N+1 query pattern on cold cache for the team list endpoint.
* refactor(proxy): extract helpers to fix PLR0915 violations
Extract `_apply_non_admin_alias_scope` from `key_aliases`,
`_resolve_team_access_group_resources` from `team_info`, and
`_enforce_list_team_v2_access` from `list_team_v2` to bring each
function under ruff's 50-statement limit. No behavior changes.
* test(ui): update tests to match new team_id / access-group signatures
- useKeyAliases, PaginatedKeyAliasSelect: add trailing `undefined` to
spy matchers for the new `team_id` param on `useInfiniteKeyAliases`
and `keyAliasesCall`.
- EntityUsage: mock new `TeamMultiSelect` child so QueryClientProvider
is not required for team-entity tests.
- ModelsCell: replace the overflow-accordion test with one that
verifies the new collapse-on-`all-proxy-models` behavior (no
accordion, single badge).
* fix(ui): send null (not '') for cleared organization_id on team update
AntD <Select allowClear> returns undefined when the user clears the
selection. Coalescing to "" caused the team-update payload to carry
organization_id: "" instead of null, relying on the backend to coerce
it. Send null directly so the intent is explicit at the source.
* poetry
* chore: regen poetry.lock for litellm-proxy-extras 0.4.64 bump
* chore: update Next.js build artifacts (2026-04-04 17:55 UTC, node v22.16.0)
---------
Co-authored-by: shivam <shivam@uni.minerva.edu>
Co-authored-by: Ryan Crabbe <ryan@berri.ai>
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
* Tag query fix (#25094)
* feat(tag-spend): implement separate scheduler job for daily tag spend updates
* fix(docker): add g++ to build dependencies in Dockerfile
* initial test cases. TODO: check scheduler init and test cases in proxy_server related to it
* resolved QPS issue when redis transaction buffer is enabled
* resolving circular import error flagged by greptile
* fix(mypy): use Optional[str] for api_base in PydanticAI provider to match superclass signature
---------
Co-authored-by: Shivam Rawat <shivam@berri.ai>
Co-authored-by: shivam <shivam@uni.minerva.edu>
Co-authored-by: Ryan Crabbe <ryan@berri.ai>
Co-authored-by: yuneng-jiang <yuneng@berri.ai>
Co-authored-by: Harish <harishgokul01@gmail.com>
Co-authored-by: Ishaan Jaffer <ishaan@berri.ai>
* Litellm ishaan march23 - MCP Toolsets + GCP Caching fix (#25146)
* feat(mcp): MCP Toolsets — curated tool subsets from one or more MCP servers (#24335)
* feat(mcp): add LiteLLM_MCPToolsetTable and mcp_toolsets to ObjectPermissionTable
* feat(mcp): add prisma migration for MCPToolset table
* feat(mcp): add MCPToolset Python types
* feat(mcp): add toolset_db.py with CRUD helpers for MCPToolset
* feat(mcp): add toolset CRUD endpoints to mcp_management_endpoints
* fix(mcp): skip allow_all_keys servers when explicit mcp_servers permission is set (toolset scope fix)
* feat(mcp): add _apply_toolset_scope and toolset route handling in server.py
* fix(mcp): resolve toolset names in responses API before fetching tools
* feat(mcp): add mcp_toolsets field to LiteLLM_ObjectPermissionTable type
* feat(mcp): register LiteLLM_MCPToolsetTable in prisma client initialization
* feat(mcp): validate mcp_toolsets in key-vs-team permission check
* feat(mcp): register toolset routes in proxy_server.py
* feat(mcp): add MCPToolset and MCPToolsetTool TypeScript types
* feat(mcp): add fetchMCPToolsets, createMCPToolset, updateMCPToolset, deleteMCPToolset API functions
* feat(mcp): add useMCPToolsets React Query hook
* feat(mcp): add toolsets (purple) as third option type in MCPServerSelector
* feat(mcp): extract toolsets from combined MCP field in key form
* feat(mcp): extract toolsets from combined MCP field in team form
* feat(mcp): show toolsets section in MCPServerPermissions read view
* feat(mcp): pass mcp_toolsets through object_permissions_view
* feat(mcp): add MCPToolsetsTab component for creating and managing toolsets
* feat(mcp): add Toolsets tab to mcp_servers.tsx
* feat(mcp): pass mcpToolsets to playground chat and responses API calls
* feat(mcp): generate correct server_url for toolsets in playground API calls
* docs(mcp): add MCP Toolsets documentation
* docs(mcp): add mcp_toolsets to sidebar
* fix(mcp): replace x-mcp-toolset-id header with ContextVar to prevent client forgery
* fix(mcp): use ContextVar + StreamingResponse for toolset MCP routes (fixes SSE streaming)
* fix(mcp): cache toolset permission lookups to avoid per-request DB calls
* test(mcp): add tests for toolset scope enforcement, ContextVar isolation, and access control
* fix(mcp): cache toolset name lookups in MCPServerManager to avoid per-request DB calls
* fix(mcp): prevent body_iter deadlock + use cached toolset lookup in responses API
- _stream_mcp_asgi_response: add done callback to handler_task that puts
the EOF sentinel on body_queue when the task exits, preventing body_iter
from hanging forever if the handler raises after headers are sent.
- litellm_proxy_mcp_handler: replace raw get_mcp_toolset_by_name() DB call
with global_mcp_server_manager.get_toolset_by_name_cached() so toolset
resolution uses the 60s TTL cache added for this purpose instead of
hitting the DB on every responses-API request.
* fix(mcp): toolset access control, asyncio fix, and real unit tests
- server.py: _apply_toolset_scope now enforces that non-admin keys must
have the requested toolset_id in their mcp_toolsets grant list;
admin keys always bypass the check.
- mcp_management_endpoints.py: three access-control fixes:
* fetch_mcp_toolsets: non-admin keys with mcp_toolsets=None now
return [] instead of all toolsets (only admins get 'all' when
the field is absent)
* fetch_mcp_toolset: non-admin keys that haven't been granted the
requested toolset_id now get 403 instead of the full result
* add_mcp_toolset: duplicate toolset_name now returns 409 Conflict
instead of an opaque 500
- proxy_server.py: use asyncio.get_running_loop() instead of
get_event_loop() inside an already-running coroutine (Python 3.10+).
- test_mcp_toolset_scope.py: replace four hollow tests that only
asserted local variable properties with real tests that call the
production fetch_mcp_toolsets() and handle_streamable_http_mcp()
functions with mocked dependencies.
* fix(mcp): add mcp_toolsets to ObjectPermissionBase, fix multi-toolset overwrite, fix delete 404, allow standalone key toolsets
* fix(mcp): add auth check on toolset resolution in responses API; union mcp_servers in _merge_toolset_permissions
* fix(mcp): handle RecordNotFoundError in update_mcp_toolset; union direct servers with toolset servers
* fix(mcp): use _user_has_admin_view; deny None mcp_toolsets for non-admin; use direct RecordNotFoundError import; fix docstring
* fix(mcp): add @default(now()) to MCPToolsetTable.updated_at; fix test for non-admin toolset access
* fix: use UniqueViolationError import; guard _ensure_eof for error/cancel only
* fix(mcp): preserve mcp_access_groups in toolset scope, use shared Redis cache for toolset perms
- Remove mcp_access_groups=[] from _apply_toolset_scope (server.py) and the
responses API toolset path (litellm_proxy_mcp_handler.py). A key's access-group
grants remain valid even when the request is scoped to a single toolset; clearing
them silently revoked legitimate entitlements.
- Switch resolve_toolset_tool_permissions and get_toolset_by_name_cached to use
user_api_key_cache (Redis-backed DualCache in production) instead of per-instance
in-memory dicts. Cache entries are now shared across workers, eliminating the
per-worker stale-toolset-permission window flagged as a P1 by Greptile.
- Use union merge (set union of tool names per server) when applying toolset
permissions in the responses API path so direct-server tool restrictions are not
overwritten by toolset permissions.
* fix(mcp): return 404 when edit_mcp_toolset target does not exist
* fix(mcp): align mcp_toolsets default to None in LiteLLM_ObjectPermissionTable
* fix(mcp): admin toolset visibility, in-place tool name mutation, test helper coercion
* fix(mcp): treat None/[] team mcp_toolsets as no restriction in key validation
* fix(mcp): allow_all_keys backward compat, blocked_tools API write-path, efficient startup query
* fix(mcp): use _mcp_active_toolset_id ContextVar to detect toolset scope, avoiding DB-default false-positive
* fix(mcp): remove dead toolset cache stubs, log invalidation failures, align schema updated_at defaults
* fix(mcp): deserialise MCPToolset from Redis cache hit, replace fastapi import in test
* fix(mcp): evict name-cache on toolset mutation, 409 on rename conflict, warning-level list errors
* fix(redis): regenerate GCP IAM token per connection for async cluster (#24426)
* fix(redis): regenerate GCP IAM token per connection for async cluster clients
Async RedisCluster was generating the IAM token once at startup and
storing it as a static password. After the 1-hour GCP token TTL, any
new connection (including to newly-discovered cluster nodes) would fail
to authenticate.
Fix: introduce GCPIAMCredentialProvider that implements redis-py's
CredentialProvider protocol. It calls _generate_gcp_iam_access_token()
on every new connection, matching what the sync redis_connect_func
already does. async_redis.RedisCluster accepts a credential_provider
kwarg which is invoked per-connection.
* refactor(redis): move GCPIAMCredentialProvider to its own file
Extract GCPIAMCredentialProvider and _generate_gcp_iam_access_token
into litellm/_redis_credential_provider.py. _redis.py imports them
from there, keeping the public API unchanged.
* fix: address Greptile review issues
- GCPIAMCredentialProvider now inherits from redis.credentials.CredentialProvider
so redis-py's async path calls get_credentials_async() properly
- move _redis_credential_provider import to top of _redis.py (PEP 8)
- remove dead else-branch that silently no-oped (gcp_service_account from
redis_kwargs.get() was always None since it's popped by _get_redis_client_logic)
- remove mid-function 'from litellm import get_secret_str' inline import
- remove unused 'call' import from test_redis.py
* chore: retrigger CI/review
* chore: sync schema.prisma copies from root
* chore: sync schema.prisma copies from root
* fix(proxy_server): use bounded asyncio.Queue with maxsize to prevent unbounded growth
* fix(a2a/pydantic_ai): make api_base Optional to match base class signature
* fix(a2a/pydantic_ai): make api_base Optional in handler and guard against None
* fix(mcp): remove unused get_all_mcp_servers import
* fix(mcp): remove unused MCPToolset import
* refactor(mcp): extract toolset permission logic to reduce statement count below PLR0915 limit
* fix(tests): update reload_servers_from_database tests to mock prisma directly
---------
Co-authored-by: Ishaan Jaff <ishaanjaffer0324@gmail.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* fix(toolset_db): lazy-import prisma to avoid ImportError when prisma not installed
* fix(tests): update UI tests for toolset tab and updated empty state text
* fix(tests): add get_mcp_server_by_name to fake_manager stub
---------
Co-authored-by: Ishaan Jaff <ishaanjaffer0324@gmail.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
* fix(pricing): add unversioned vertex_ai/claude-haiku-4-5 entry
Missing unversioned entry causes cost tracking to return $0.00 for
all requests using vertex_ai/claude-haiku-4-5. All other Vertex AI
Claude models have both versioned and unversioned entries.
* fix(router): skip misleading tags error when no candidates (e.g. cooldown)
Return early from get_deployments_for_tag when healthy_deployments is empty so
tag-based routing does not raise no_deployments_with_tag_routing after cooldown
filters all deployments. Adds regression test.
Made-with: Cursor
* feat(oci): add embedding support and update model catalog
- Add OCIEmbeddingConfig for OCI GenAI embedding models
- Add 16 new chat models (Cohere, Meta Llama, xAI Grok, Google Gemini)
- Add 8 embedding models (Cohere embed v3.0, v4.0)
- Update documentation with embedding examples
- Update pricing for all new models
* test(oci): add unit tests for OCI embedding support
- 17 unit tests covering OCIEmbeddingConfig
- Tests for URL generation, param mapping, request/response transform
- Tests for model pricing JSON completeness
* style(oci): format with black and ruff
* fix(oci): correct embedding request body format
OCI embedText API expects inputs, truncate, and inputType at the
top level of the request body, not nested under embedTextDetails.
Fixed transformation and updated tests accordingly.
Verified with real OCI API: 3/3 embedding models working.
* docs: clarify tag routing early return and test intent
Made-with: Cursor
* fix(oci): address code review findings from Greptile
- P1: Fix signing URL mismatch with custom api_base by accepting
api_base parameter in transform_embedding_request
- P2: Remove encoding_format from supported params (OCI does not
support it, was silently dropped)
- P2: Raise ValueError for token-array inputs instead of silently
converting to string representation
- Add test for token-list rejection
* fix(mcp): add STS AssumeRole support for MCP SigV4 authentication
MCPSigV4Auth only supported static AWS credentials or the boto3 default
credential chain. Production Kubernetes environments typically authenticate
via IAM role assumption (sts:AssumeRole), which was not possible.
Add aws_role_name and aws_session_name parameters to the MCP SigV4 auth
stack. When aws_role_name is provided, MCPSigV4Auth calls sts:AssumeRole
to obtain temporary credentials before signing requests. Explicit keys,
if also provided, are used as the source identity for the STS call;
otherwise ambient credentials (pod role, instance profile) are used.
* fix: stop logging credential values and add missing redaction patterns
Replaces raw credential values in debug/error log messages with
boolean presence checks or type names. Adds PEM block, GCP token,
JWT, SAS token, and service-account blob patterns to the redaction
filter. Fixes private_key pattern to capture full PEM blocks instead
of stopping at the first whitespace.
Addresses: Vertex AI credential JSON (including RSA private key)
being logged to stderr on health check failures.
* fix: log only field names for UserAPIKeyAuth, not full object
* style: apply black formatting to experimental_mcp_client/client.py
* style: fix black/isort formatting and mypy error in proxy_server.py
- Fix black formatting in experimental_mcp_client/client.py (done in prev commit)
- Fix black/isort formatting in key_management_endpoints.py, proxy_server.py, transformation.py
- Fix mypy: iterate over optional list safely (access_group_ids or []) in proxy_server.py
* fix(test): patch check_migration.verbose_logger directly to fix xdist ordering issue
When test_proxy_cli.py tests run before test_check_migration.py in the same
xdist worker, litellm.proxy.db.check_migration is already in sys.modules.
Patching litellm._logging.verbose_logger has no effect on the already-bound
reference. Patch the correct target (check_migration.verbose_logger) and
import the module before patching so the order doesn't matter.
* fix(mypy): make api_base Optional in PydanticAIProviderConfig to match base class signature
---------
Co-authored-by: Ihsan Soydemir <soydemir.ihsan@gmail.com>
Co-authored-by: Milan <milan@berri.ai>
Co-authored-by: Daniel Gandolfi <danielgandolfi@gmail.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: michelligabriele <gabriele.michelli@icloud.com>
Co-authored-by: user <70670632+stuxf@users.noreply.github.com>
Co-authored-by: Ishaan Jaffer <ishaanjaffer0324@gmail.com>
* feat: add brave/search to model_prices_and_context_window.json (#25042)
Brave Search is supported by litellm as a search provider (documented at
docs.litellm.ai/docs/search/brave and listed in provider_endpoints_support.json)
but was missing from model_prices_and_context_window.json, making it invisible
to any code that discovers search providers from litellm.model_cost.
Cost: $0.005/query ($5 per 1,000 requests) per https://brave.com/search/api/
* feat(models): add NVIDIA Nemotron 3 Super 120B on Bedrock (#24588)
* feat(models): add NVIDIA Nemotron 3 Super 120B on Bedrock
Add model definition for nvidia.nemotron-3-super-120b-a12b-v1 via
Bedrock Converse API with pricing, context window (256k/32k), and
capability flags (function calling, tool choice, system messages).
* fix model ID to nvidia.nemotron-super-3-120b + add tests
Correct the Bedrock model ID from nvidia.nemotron-3-super-120b-a12b-v1
(NVIDIA's internal name) to nvidia.nemotron-super-3-120b (the actual
AWS Bedrock programmatic model ID). Add unit tests verifying model
resolution, pricing, and context window.
* fix(proxy): allow JWT auth for /v1/mcp/server sub-paths (#24698)
mcp_routes only contained "/v1/mcp/server" (exact match). Starlette's
compile_path produces an end-anchored regex, so sub-paths like
/register, /health, /submissions, /oauth/* all failed the JWT
allowed_routes_check. Add a {path:path} wildcard entry so all
sub-paths are covered.
---------
Co-authored-by: Daniel Yudelevich <4537920+yudelevi@users.noreply.github.com>
Co-authored-by: michelligabriele <gabriele.michelli@icloud.com>
* add DD Tracing (#24033)
* feat(models): add Azure GPT-5.4 mini and nano variants (#24045)
Add `azure/gpt-5.4-mini` and `azure/gpt-5.4-nano` to the model
database with official pricing from Azure OpenAI:
- GPT-5.4 mini: $0.75/M input, $0.075/M cached, $4.5/M output
- GPT-5.4 nano: $0.20/M input, $0.02/M cached, $1.25/M output
Both models support:
- 1.05M input / 128K output context window
- Chat, batch, and responses endpoints
- Function calling, tools, vision, reasoning
- Prompt caching with automatic tiered pricing
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* Add new model pricing details for volcengine Doubao-Seed-2.0 series (#23871)
Add entries for volcengine Doubao-Seed-2.0 series
* fix(mcp): support refresh_token grant type in OAuth token endpoint (#23701)
* fix(mcp): support refresh_token grant type in OAuth token endpoint (#23700)
The .well-known/oauth-authorization-server metadata advertises
refresh_token as a supported grant type, but the token endpoint
rejected it with HTTP 400. This adds refresh_token grant support
so MCP clients can refresh expired tokens without re-authenticating.
* test(mcp): add tests for refresh_token grant type in OAuth token endpoint
* fix(mcp): move code_verifier guard into authorization_code branch
code_verifier is only relevant for authorization_code grants (PKCE).
Move it inside the else branch so it doesn't apply to refresh_token.
* fix(mcp): guard None client_secret and forward scope in token exchange
- Conditionally include client_secret in form data to prevent httpx
from sending the literal string "None" (applies to both
authorization_code and refresh_token branches)
- Forward optional scope parameter per RFC 6749 §6, allowing clients
to request a subset of originally-granted scopes on refresh
* fix(mcp): validate code param in authorization_code grant
Guard against None code being form-encoded as literal string "None"
by httpx, symmetric with the existing refresh_token guard.
* docs: add incident report for guardrail logging secret exposure (#24059)
Add blog post documenting the guardrail logging path exposing internal
request data (e.g. Authorization headers) in spend logs and OTEL traces.
Fix available in LiteLLM 1.82.3+.
Made-with: Cursor
* [Fix] Datadog LLM Observability tags format (env, service, version missing) (#23673)
* tag fix
* greptile comment
* fix(ci): stabilize 6 failing CI jobs
1. mypy: remove duplicate type annotation for token_data in discoverable_endpoints.py
2. integrations tests: add parameterized to CI test deps
3. doc quality: document OTEL_IGNORE_CONTEXT_PROPAGATION env key
4. security: allowlist CVE-2026-2673, CVE-2026-3644, CVE-2026-4224 (no fix available)
5. proxy_store_model_in_db: fix missing x-litellm-call-id header on error responses
6. google tests: add --retries 3 for transient Vertex AI rate limits
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix(streaming): handle RuntimeError during model_copy in streaming handler
The race condition occurs when model_copy(deep=True) tries to deepcopy
_hidden_params dict while it's being concurrently modified by logging
callbacks. Fall back to shallow copy if the deep copy fails.
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
* fix(cost): handle non-string traffic_type in cost calculator + add retries
1. Fix AttributeError in _map_traffic_type_to_service_tier when traffic_type
is an integer (cast to str before calling .upper()). This was causing
pass-through vertex spend logging to fail silently.
2. Add --retries to llm_translation_testing for flaky external API calls.
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
---------
Co-authored-by: Emerson Gomes <emerson.gomes@thalesgroup.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: ExMatics HydrogenC <33123710+HydrogenC@users.noreply.github.com>
Co-authored-by: Jack Venberg <jack.venberg@rover.com>
Co-authored-by: milan-berri <milan@berri.ai>
Co-authored-by: Shivam Rawat <161387515+shivamrawat1@users.noreply.github.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Ishaan Jaff <ishaan-jaff@users.noreply.github.com>
