Commit Graph

3335 Commits

Author SHA1 Message Date
ryan-crabbe-berri
65d9fadf45
Merge pull request #25575 from BerriAI/litellm_feat-per-guardrail-opt-out-for-global-guardrails
feat(guardrails): per-team opt-out for specific global guardrails
2026-04-13 13:31:23 -07:00
Ryan Crabbe
2d14e4a4ed
test(ui/team): fix guardrails overview test for new component
Updates the expected header text to "Guardrails Settings" to match
GuardrailSettingsView's rendering, and moves the mock guardrails
from team_info.guardrails (legacy top-level path that nothing
reads) to team_info.metadata.guardrails where the component
actually looks. Also tightens the assertion to verify the
individual guardrail names appear, not just the section header.
2026-04-13 12:20:30 -07:00
Ryan Crabbe
842523a918
chore(ui): use antd in GuardrailSettingsView and document the rule
Converts GuardrailSettingsView from @tremor/react (Badge, Text) to
antd (Tag, plain spans) as part of the Tremor migration. Also
captures the "no new Tremor imports" rule in CLAUDE.md and expands
the existing note in AGENTS.md with the specific antd equivalents
and the yellow→gold gotcha.
2026-04-13 11:49:39 -07:00
Ryan Crabbe
84d7816bc9
refactor(ui/team): extract GuardrailSettingsView and reuse across team views
Pulls the Global / Team-specific subsection rendering out of
TeamInfo.tsx into a shared GuardrailSettingsView component with
card and inline variants, used on both the team Overview tab
(inside the existing Tremor Card) and the Team Settings tab read
view. The Global subsection header now carries a GlobalOutlined
icon, and since the icon is load-bearing the edit-form chip
coloring is simplified to a single blue instead of green/blue.
2026-04-13 11:38:23 -07:00
Ryan Crabbe
1dcccfc1d1
feat(ui/team): show guardrails in team settings read view
The Team Settings tab's read view listed every team field except
guardrails. Adds a Guardrails entry after Status with the same
Global / Team-specific subsections used on the Overview tab, so
the kill switch state and per-section membership are visible
without entering edit mode.
2026-04-13 11:08:39 -07:00
Ryan Crabbe
6ae693b85c
refactor(ui/team): split team read view guardrails into Global and Team-specific sections
Replaces the flat guardrails list with two subsections under the
Guardrails card, so the global vs. team-specific distinction is
carried by the section headers instead of per-badge markers. The
kill-switch state now renders in place of the Global subsection as
"Bypassed for this team", and the separate "Disable Global
Guardrails" field with its confusing "Disabled - Global guardrails
active" badge is removed.
2026-04-13 10:57:10 -07:00
Ryan Crabbe
1f57292363
fix(ui/team): add GlobalOutlined icon to global guardrail markers
Addresses a11y feedback — global vs. non-global guardrails were
distinguished only by color (green vs. blue). Adds GlobalOutlined
next to global guardrails in (1) the selected-chip tagRender, (2)
the dropdown OptGroup label, and (3) the team info read view badge.
2026-04-13 10:55:09 -07:00
Sameer Kankute
fa605d85c0
Merge pull request #25616 from BerriAI/main
merge main
2026-04-13 08:43:43 +05:30
Yuneng Jiang
1857be43a7
Merge remote-tracking branch 'origin/main' into litellm_regen_key_modal_antd 2026-04-11 20:45:17 -07:00
Yuneng Jiang
7c6bd98fdf
fix: address PR review comments on RegenerateKeyModal
- Remove redundant useEffect cleanup that duplicated handleClose logic
- Remove unnecessary currentAccessToken state, use accessToken from hook directly
2026-04-11 18:01:02 -07:00
Ryan Crabbe
30c2357fe3
fix(ui/team): annotate effectiveGuardrails as string[] for strict typecheck
The new effectiveGuardrails computation derived its element type from
`info.metadata?.guardrails`, which is implicitly typed `any[]`, so the
downstream `.map((name) => ...)` callback inferred `name` as implicit
`any` and broke the production typecheck.
2026-04-11 17:48:16 -07:00
Ryan Crabbe
3d72e2a6f3
fix(guardrails): address PR #25575 review feedback + sync kill switch with opt-out list
Renames the new per-guardrail opt-out field from `disabled_global_guardrails`
to `opted_out_global_guardrails` to eliminate the one-character collision with
the legacy `disable_global_guardrails` boolean kill switch. Adds a type guard
on the new gate so a misnamed bool can't crash the guardrail check. Filters
duplicates out of the team-edit guardrail display for legacy teams that have a
global name persisted in `metadata.guardrails` from before this PR. Drops the
unused `isGuardrailsLoading` and `guardrailsError` destructures left in
AddModelForm after the hook refactor.

Adds Python tests for the new gate behavior (root, litellm_metadata, metadata,
non-matching name, empty list, malformed bool value, opt-in coexistence) and
extends useGuardrails.test.ts to exercise the global / optional partition
logic that the rebuilt hook performs in its `select` transform.

Wires the legacy kill switch and the new opt-out list together in the team
edit form so they can never fall out of sync:

- Toggling the kill switch reactively updates the Guardrails Select via
  `onValuesChange` — switch on strips all globals from the selection, switch
  off re-adds them. Existing opt-in extras are preserved either way.
- When the switch is on, global options in the Select are individually
  disabled (greyed out) so the user can still manage opt-in guardrails but
  cannot accidentally re-enable a global the kill switch is bypassing.
- The save handler writes both fields together: `disable_global_guardrails`
  reflects the switch, and `opted_out_global_guardrails` is set to either
  every global (when the switch is on) or the user's explicit opt-outs.
- `effectiveGuardrails` for the form's initialValues honors the kill switch
  on legacy teams so the form opens in a state that matches what the runtime
  gate is actually doing — fixes the visual lie where chips appeared active
  while the switch was bypassing them.

The backend gate already reads the list as the primary path with the bool
as a fallback, so untouched legacy teams keep working until they get edited,
at which point they migrate naturally.
2026-04-11 17:48:16 -07:00
Ryan Crabbe
59f66af0f9
fix(ui/team): wait for useGuardrails to resolve before rendering edit form
The team edit form computed its `guardrails` initialValues from
`globalGuardrailNames`, which is empty until `useGuardrails` resolves.
Because Ant Design's `<Form initialValues>` is read once on mount, a
user who clicked Edit before the query resolved would see globals
missing from the Select and could save that stale state, silently
opting the team out of every global guardrail.

Gate the form on `!isGuardrailsLoading` so initialValues always sees
the resolved set.
2026-04-11 17:48:16 -07:00
Ryan Crabbe
f9639654fa
feat(ui/team): per-guardrail opt-out for global guardrails
Replace the team info page's tags-mode guardrails Select with a grouped
multi-select that splits guardrails into "Global" (default_on=true) and
"Other" sections. On submit, derive both metadata fields from the single
selection array:

  - metadata.guardrails: non-global opt-ins (additive)
  - metadata.disabled_global_guardrails: globals the team has opted out of

The legacy disable_global_guardrails kill-switch toggle stays alongside
the multiselect and remains semantically distinct: it kills all global
guardrails including any added in the future, while the new list lets
new globals auto-apply unless explicitly excluded.

Other changes:
  - Switch from a manual useEffect/useState guardrails fetch to the
    consolidated useGuardrails hook
  - Replace stale "Select existing guardrails or enter new ones" help
    text with a single tooltip on each section's info icon
  - Render selected chips with green/blue color coding via tagRender so
    global vs non-global is visible without opening the dropdown
  - Drop the redundant [Global] tag from inside the OptGroup options
    (the group label already conveys it)
  - Update the read-only display to show the effective set (globals not
    opted out + additive opt-ins) with [Global] suffix on globals
2026-04-11 17:48:16 -07:00
Ryan Crabbe
a9d64f8620
refactor(ui/guardrails): expose full data from useGuardrails hook
Extend useGuardrails to return the full guardrail objects plus derived
globalGuardrailNames / optionalGuardrailNames sets via React Query's
select option, instead of just an array of names. Update its existing
consumer (AddModelForm) to extract names from the new shape.

The previous shape was tailored to AddModelForm's single use case
(populate a Select with names). The team info per-guardrail opt-out
work needs default_on per guardrail to split globals from non-globals,
which the old shape couldn't provide. Consolidating into the existing
hook gives both consumers one source of truth and one React Query
cache entry instead of two parallel fetches.

- useGuardrails.ts: rewrite return type, derive global/optional sets
  in select(); preserve the existing query key and auth-gate semantics
- AddModelForm.tsx: extract names from data?.guardrails.map(...)
- AddModelForm.test.tsx: update mock to return the new shape (also
  fixes a pre-existing shape mismatch in the mock)
- useGuardrails.test.ts: update 3 assertions to read names via
  data?.guardrails.map(...) instead of asserting against the flat array
2026-04-11 17:48:16 -07:00
Yuneng Jiang
909247785e
Merge remote-tracking branch 'origin' into litellm_internal_staging_04_11_2026 2026-04-11 15:41:03 -07:00
yuneng-jiang
56e82451df
Merge pull request #25458 from BerriAI/litellm_team_members_logs
Team member permission /spend/logs for team-wide spend logs (UI + RBAC)
2026-04-11 13:55:00 -07:00
Sameer Kankute
c13be44e44
feat(guardrails): optional skip system message in unified guardrail inputs (#25481)
* feat(guardrails): optional skip system message in unified guardrail inputs

Made-with: Cursor

* feat(dashboard): skip_system_message_in_guardrail in guardrail UI

Add a tri-state control (inherit / yes / no) when creating or editing
guardrails so admins can set litellm_params.skip_system_message_in_guardrail
without YAML. Table edit merges existing litellm_params before PUT to avoid
wiping content-filter and other provider fields.

Document the dashboard flow in the guardrails quick start with a screenshot.

Made-with: Cursor

* fix(guardrails): type structured_messages as AllMessageValues for mypy

Use AllMessageValues in openai_messages_without_system and cast adapter
request messages so GenericGuardrailAPIInputs matches TypedDict.

Made-with: Cursor
2026-04-11 08:53:24 -07:00
Yuneng Jiang
a771e1939c
Fix unhandled "window is not defined" error in user_edit_view tests
Tremor's internal Tooltip component sets a setTimeout that fires after
the jsdom test environment tears down, causing a ReferenceError. Add
afterEach that flushes pending timers before cleanup.
2026-04-10 00:10:40 -07:00
Yuneng Jiang
2e0af3795a
fix(e2e): broaden Copy Key button regex to match both modal versions
On case-sensitive Linux CI, the old regenerate_key_modal.tsx from main
can coexist with the new RegenerateKeyModal.tsx after merge. The old
modal renders "Copy Virtual Key" while the new one renders "Copy Key".
Use /Copy.*Key/ to match both.
2026-04-09 23:36:35 -07:00
Yuneng Jiang
89320a955c
fix(e2e): remove flaky banner check and increase regenerate key timeout 2026-04-09 23:21:01 -07:00
Yuneng Jiang
9071dbba12
fix(ui): remove leftover setIsOwnKey call after state removal 2026-04-09 21:24:22 -07:00
yuneng-jiang
cc43d09d79
Potential fix for pull request finding 'CodeQL / Unused variable, import, function or class'
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
2026-04-09 21:19:02 -07:00
Yuneng Jiang
71640f062c
Merge remote-tracking branch 'origin/main' into litellm_regen_key_modal_antd 2026-04-09 21:14:57 -07:00
Yuneng Jiang
92dbd2c491
address greptile review feedback (greploop iteration 1)
Remove leftover 10000ms per-test timeout in add_model_tab.test.tsx that was
missed in the initial sweep. The test now inherits the 30000ms global.
2026-04-09 20:54:36 -07:00
Yuneng Jiang
3a316b9131
[Test] UI - Unit tests: raise global vitest timeout and remove per-test overrides
Raise vitest testTimeout from 10s to 30s and drop per-test timeout overrides
across UI unit tests. Group CreateUserButton and TeamInfo tests under nested
describe blocks to make the most flaky suites easier to scan.
2026-04-09 20:42:30 -07:00
Yuneng Jiang
1d50f774e2
fix(ui): support all duration suffixes in regenerate expiry preview
calculateNewExpiryTime only handled s/h/d, but the grace-period
validation and backend accept m, w, and mo as well. Entering any of
those in the Expire Key field caused the function to return null,
which then propagated as expires: null in the onKeyUpdate payload —
the parent UI would then render the expiry as "Never" even though
the backend had correctly applied the new expiry.

Extend the suffix check to cover s/m/h/d/w/mo, matching "mo" before
"m" so "1mo" isn't misread as minutes. Also nullish-coalesce the
call site so an unparseable duration falls back to the previous
expiry instead of null. Add parametric tests for each supported
suffix plus a regression test for the null fallback.
2026-04-09 20:31:35 -07:00
Yuneng Jiang
f95ef935ef
fix(ui): prefer form values over API echo in regenerate update payload
The regenerate endpoint returns a GenerateKeyResponse that inherits
max_budget/tpm_limit/rpm_limit from KeyRequestBase, so the API echoes
the existing values back. The previous updatedKeyData layout spread
...response *after* the explicit formValues assignments, which meant
the user's just-submitted edits were silently overwritten by the API
echo before being propagated to the parent via onKeyUpdate.

Reorder so the response spread comes first and the formValues-derived
fields override it, and add a regression test that mocks a response
with stale limits to lock the behavior in. Also drop the two leftover
debug console.log statements.
2026-04-09 20:25:10 -07:00
Yuneng Jiang
839d9bd5f3
refactor(ui): polish regenerate key success view
- Label the key block with a small "Virtual Key" caption so the gray
  box is clearly the key container.
- Move the Copy Key action to the modal footer as a primary button
  with icon; inline copy icon next to the key is removed.
- Swap the button to "Copied" with a check icon on success instead of
  firing a notification — less noisy and keeps feedback in place.
- Disable clicking outside the modal to close (maskClosable=false) so
  users must explicitly dismiss via Close or X.
- Enlarge the key text and let its container span the full modal
  width.
- Tests updated accordingly, including a new test for the copied-state
  swap and the "Virtual Key" label.
2026-04-09 20:14:01 -07:00
yuneng-jiang
ce75598f97
Merge pull request #25384 from BerriAI/litellm_/bold-pare
[Fix] UI: improve storage handling and Dockerfile consistency
2026-04-09 19:55:24 -07:00
Yuneng Jiang
15f7cc9134
refactor(ui): replace success-view divs in regenerate key modal with antd
Use Flex, Typography.Paragraph (with copyable), and Typography.Text
instead of raw divs + code block + CopyToClipboard wrapper. Drops the
direct react-copy-to-clipboard dependency in this component in favor
of antd's native copyable support.

Also fixes two test issues surfaced when running the e2e locally:
- RegenerateKeyModal.test.tsx no longer mocks react-copy-to-clipboard
  (the component no longer imports it), removing the CJS require()
  inside an ESM mock factory flagged by Greptile.
- keys.spec.ts scopes the Regenerate and Copy lookups to the modal.
  The Regenerate button has an icon whose aria-label ("sync") is
  concatenated into the button's accessible name, so an exact-match
  lookup on "Regenerate" failed; and the new Paragraph copyable
  renders a generic "Copy" button that collided with the other
  copyable fields on the key info view.
2026-04-09 18:43:28 -07:00
shivam
31f750146b
added option to allow team user to see logs of team 2026-04-09 17:43:20 -07:00
Chetan Soni
ce2add3b16 feat(mcp): add per-user OAuth token storage for interactive MCP flows 2026-04-09 12:42:42 -07:00
Abhijoy Sarkar
c688d9d6bc
Add PromptGuard guardrail integration (#24268)
* Add PromptGuard guardrail integration

Add PromptGuard as a first-class guardrail vendor in LiteLLM's proxy,
supporting prompt injection detection, PII redaction, topic filtering,
entity blocklists, and hallucination detection via PromptGuard's
/api/v1/guard API endpoint.

Backend:
- Add PROMPTGUARD to SupportedGuardrailIntegrations enum
- Implement PromptGuardGuardrail (CustomGuardrail subclass) with
  apply_guardrail handling allow/block/redact decisions
- Add Pydantic config model with api_key, api_base, ui_friendly_name
- Auto-discovered via guardrail_hooks/promptguard/__init__.py registries

Frontend:
- Add PromptGuard partner card to Guardrail Garden with eval scores
- Add preset configuration for quick setup
- Add logo to guardrailLogoMap

Tests:
- 30 unit tests covering configuration, allow/block/redact actions,
  request payload construction, error handling, config model, and
  registry wiring

* Fix redact path and init ordering per review feedback

- P1: Update structured_messages (not just texts) when PromptGuard
  returns a redact decision, so PII redaction is effective for the
  primary LLM message path
- P2: Validate credentials before allocating the HTTPX client so
  resources aren't acquired if PromptGuardMissingCredentials is raised
- Add tests for structured_messages redaction and texts-only redaction

* Harden PromptGuard integration: fail-open, event hooks, images, docs

- Add block_on_error config (default fail-closed, configurable fail-open)
- Declare supported_event_hooks (pre_call, post_call) like other vendors
- Forward images from GenericGuardrailAPIInputs to PromptGuard API
- Wrap API call in try/except for resilient error handling
- Add comprehensive documentation page with config examples
- Register docs page in sidebar alongside other guardrail providers
- Expand test suite from 32 to 40 tests covering new functionality

* Fix dict[str, Any] -> Dict[str, Any] for Python 3.8 compat

* Address remaining Greptile feedback: timeout, redact guard

- Add explicit 10s timeout to async_handler.post() to prevent
  indefinite hangs when PromptGuard API is unresponsive
- Guard redact path: only update inputs["texts"] when the key
  was originally present, avoiding phantom key injection
- Add test: redact with structured_messages only does not create
  texts key (41 tests total)

* Fix CI lint: black formatting, add PromptGuardConfigModel to LitellmParams

- Reformat promptguard.py to match CI black version (parenthesization)
- Add PromptGuardConfigModel as base class of LitellmParams for proper
  Pydantic schema validation, consistent with all other guardrail vendors
- Use litellm_params.block_on_error directly (now a typed field)

* Address Greptile review: redact path, null decision, error context

- P1: Filter _extract_texts_from_messages to user-role messages only,
  preventing system/assistant content from being injected into texts
- P1: Strengthen test_redact_updates_structured_messages assertion from
  weak `in` check to strict equality, catching the injection bug
- P2: Use `result.get("decision") or "allow"` to handle explicit null
  decision values (not just absent keys)
- P2: Wrap bare exception re-raise in GuardrailRaisedException so the
  caller knows which guardrail failed (block_on_error=True path)
- P2: Add static Promptguard entry in guardrail_provider_map so the
  preset works before populateGuardrailProviderMap is called
- Add test for explicit null decision treated as allow

* Fix black formatting: collapse f-string in error message
2026-04-09 08:12:24 -07:00
Yuneng Jiang
e42baeb5ab
[Refactor] UI - Virtual Keys: migrate regenerate key modal to AntD
Replace Tremor components in the regenerate key modal with Ant Design
equivalents and move the component to a new PascalCase file. The form
layout now uses Row/Col to place Max Budget, TPM Limit, and RPM Limit
on one row and Expire Key with Grace Period on another, reducing the
vertical footprint. The success view shows an Alert banner, the key
alias as secondary context, and the regenerated key in a monospace
block with an inline primary Copy button.

Also adds unit tests for the new component and updates the existing
Playwright spec to match the new banner and button text.
2026-04-08 23:52:25 -07:00
Yuneng Jiang
20ed120d1a
[Fix] Let setSecureItem propagate storage errors to callers
Remove the silent try/catch from setSecureItem so OAuth hooks can
surface actionable "enable storage" guidance instead of a cryptic
"state lost" error after the round-trip. Add a local try/catch in
ChatUI where the storage write is non-critical.
2026-04-08 22:13:55 -07:00
Yuneng Jiang
70a5c27cbd
[Fix] Address review feedback on storage utility and Dockerfiles
- Dockerfile.health_check: HEALTHCHECK now verifies the script is intact
  instead of unconditionally exiting 0
- secureStorage.ts: replace deprecated escape/unescape with
  encodeURIComponent/decodeURIComponent; don't delete legacy values on
  decode failure so in-flight flows can time out naturally
- OAuth callback: add same-origin check before redirecting to stored
  return URL
2026-04-08 17:51:34 -07:00
Yuneng Jiang
a881ac5133
[Fix] UI: resolve CodeQL security alerts and Dockerfile.health_check hardening
Port security fixes from litellm_v1.82.3.dev.6:
- Use secureStorage (sessionStorage wrapper) instead of raw storage for tokens
- Add URL validation for stored worker URLs to prevent open redirects
- Add same-origin checks before redirecting to stored return URLs
- Harden Dockerfile.health_check with non-root user and exec-form HEALTHCHECK
2026-04-08 17:21:25 -07:00
Yuneng Jiang
467dbc4a3c
[Fix] Remove old broken key tests superseded by proxy-admin/keys.spec.ts 2026-04-08 13:32:37 -07:00
Yuneng Jiang
4ee7d42981
[Fix] Restructure HTML files after UI build so extensionless routes work in CI 2026-04-08 13:24:52 -07:00
Yuneng Jiang
a8f4f464ce
[Fix] Add missing test fixtures and address review feedback
- Add constants.ts with all required exports (key aliases, team IDs)
- Add fixtures/users.ts with all role definitions and storage paths
- Add fixtures/seed.sql for deterministic test database seeding
- Remove Firefox project from playwright config (only Chromium installed)
- Remove unused variable in teams.spec.ts
- Rename CircleCI job to e2e_ui_testing
2026-04-08 12:40:41 -07:00
Yuneng Jiang
d09d98a70a
[Feature] E2E UI tests: proxy-admin team and key management with CI integration
Add Playwright E2E tests covering proxy admin team and key management
workflows, with a self-contained test runner and CircleCI integration.

Tests cover: create team, invite user, edit/delete team members, create
key in team, regenerate key, update TPM/RPM limits, delete key, and
verify internal user keys are visible.

Infrastructure: run_e2e.sh builds the UI from source before starting
the proxy, ensuring tests always run against the latest UI changes.
Added data-testid attributes to key UI components for reliable selectors.
2026-04-08 11:51:15 -07:00
Yuneng Jiang
54f4be6ab6
Merge remote-tracking branch 'origin/main' into litellm_ui_e2e_psql_pr 2026-04-06 23:18:35 -07:00
yuneng-jiang
39c1042258
[Docs] Add cosign Docker image verification steps to security blog posts (#25122)
* docs(blog): add cosign Docker image verification instructions

Add steps for verifying Docker images with cosign to three security blog posts:
CI/CD v2, Security Townhall, and Security Update.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs(proxy): add cosign verification to Docker/Helm/Terraform deploy page

Add image signature verification steps to the main deployment doc so
users pulling Docker images know how to verify them with cosign.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* chore: fixes

* Update index.md

Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>

* [Docs] Scope cosign signing docs to GHCR and specify starting version

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* [Docs] Add starting version callout to ci_cd_v2 blog post

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: Krrish Dholakia <krrish+github@berri.ai>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
2026-04-06 09:59:27 -07:00
Yuneng Jiang
1d3fb58752
chore: fixes 2026-04-05 01:36:02 -07:00
ryan-crabbe-berri
6bc4b46e56
Merge pull request #25156 from BerriAI/litellm_ryan-apr-4
Litellm ryan apr 4
2026-04-04 16:56:03 -07:00
ryan-crabbe-berri
8ecbf757b2
Merge pull request #25038 from BerriAI/litellm_feat-add-guardrail
feat: allow adding team guardrails from the UI
2026-04-04 16:48:43 -07:00
Ryan Crabbe
a36fe70fde
test(ui): fix guardrails.test.tsx after antd Tabs migration
The test was hitting "No QueryClient set" because TeamGuardrailsTab
(which pulls in useRegisterGuardrail) was not mocked alongside the
other tab children. Added a mock.

Also: the "+ Add New Guardrail" assertion was silently relying on
Tremor Tabs rendering all panels at once. antd Tabs only renders
the active tab's content, and defaultActiveKey is "submitted", so
the button in the "Guardrails" tab wasn't in the DOM. Clicking the
Guardrails tab first before asserting.
2026-04-04 16:32:55 -07:00
ishaan-berri
693ad49719
Litellm ishaan march23 - MCP Toolsets + GCP Caching fix (#25146) (#25155)
* Litellm ishaan march23 - MCP Toolsets + GCP Caching fix  (#25146)

* feat(mcp): MCP Toolsets — curated tool subsets from one or more MCP servers (#24335)

* feat(mcp): add LiteLLM_MCPToolsetTable and mcp_toolsets to ObjectPermissionTable

* feat(mcp): add prisma migration for MCPToolset table

* feat(mcp): add MCPToolset Python types

* feat(mcp): add toolset_db.py with CRUD helpers for MCPToolset

* feat(mcp): add toolset CRUD endpoints to mcp_management_endpoints

* fix(mcp): skip allow_all_keys servers when explicit mcp_servers permission is set (toolset scope fix)

* feat(mcp): add _apply_toolset_scope and toolset route handling in server.py

* fix(mcp): resolve toolset names in responses API before fetching tools

* feat(mcp): add mcp_toolsets field to LiteLLM_ObjectPermissionTable type

* feat(mcp): register LiteLLM_MCPToolsetTable in prisma client initialization

* feat(mcp): validate mcp_toolsets in key-vs-team permission check

* feat(mcp): register toolset routes in proxy_server.py

* feat(mcp): add MCPToolset and MCPToolsetTool TypeScript types

* feat(mcp): add fetchMCPToolsets, createMCPToolset, updateMCPToolset, deleteMCPToolset API functions

* feat(mcp): add useMCPToolsets React Query hook

* feat(mcp): add toolsets (purple) as third option type in MCPServerSelector

* feat(mcp): extract toolsets from combined MCP field in key form

* feat(mcp): extract toolsets from combined MCP field in team form

* feat(mcp): show toolsets section in MCPServerPermissions read view

* feat(mcp): pass mcp_toolsets through object_permissions_view

* feat(mcp): add MCPToolsetsTab component for creating and managing toolsets

* feat(mcp): add Toolsets tab to mcp_servers.tsx

* feat(mcp): pass mcpToolsets to playground chat and responses API calls

* feat(mcp): generate correct server_url for toolsets in playground API calls

* docs(mcp): add MCP Toolsets documentation

* docs(mcp): add mcp_toolsets to sidebar

* fix(mcp): replace x-mcp-toolset-id header with ContextVar to prevent client forgery

* fix(mcp): use ContextVar + StreamingResponse for toolset MCP routes (fixes SSE streaming)

* fix(mcp): cache toolset permission lookups to avoid per-request DB calls

* test(mcp): add tests for toolset scope enforcement, ContextVar isolation, and access control

* fix(mcp): cache toolset name lookups in MCPServerManager to avoid per-request DB calls

* fix(mcp): prevent body_iter deadlock + use cached toolset lookup in responses API

- _stream_mcp_asgi_response: add done callback to handler_task that puts
  the EOF sentinel on body_queue when the task exits, preventing body_iter
  from hanging forever if the handler raises after headers are sent.
- litellm_proxy_mcp_handler: replace raw get_mcp_toolset_by_name() DB call
  with global_mcp_server_manager.get_toolset_by_name_cached() so toolset
  resolution uses the 60s TTL cache added for this purpose instead of
  hitting the DB on every responses-API request.

* fix(mcp): toolset access control, asyncio fix, and real unit tests

- server.py: _apply_toolset_scope now enforces that non-admin keys must
  have the requested toolset_id in their mcp_toolsets grant list;
  admin keys always bypass the check.
- mcp_management_endpoints.py: three access-control fixes:
  * fetch_mcp_toolsets: non-admin keys with mcp_toolsets=None now
    return [] instead of all toolsets (only admins get 'all' when
    the field is absent)
  * fetch_mcp_toolset: non-admin keys that haven't been granted the
    requested toolset_id now get 403 instead of the full result
  * add_mcp_toolset: duplicate toolset_name now returns 409 Conflict
    instead of an opaque 500
- proxy_server.py: use asyncio.get_running_loop() instead of
  get_event_loop() inside an already-running coroutine (Python 3.10+).
- test_mcp_toolset_scope.py: replace four hollow tests that only
  asserted local variable properties with real tests that call the
  production fetch_mcp_toolsets() and handle_streamable_http_mcp()
  functions with mocked dependencies.

* fix(mcp): add mcp_toolsets to ObjectPermissionBase, fix multi-toolset overwrite, fix delete 404, allow standalone key toolsets

* fix(mcp): add auth check on toolset resolution in responses API; union mcp_servers in _merge_toolset_permissions

* fix(mcp): handle RecordNotFoundError in update_mcp_toolset; union direct servers with toolset servers

* fix(mcp): use _user_has_admin_view; deny None mcp_toolsets for non-admin; use direct RecordNotFoundError import; fix docstring

* fix(mcp): add @default(now()) to MCPToolsetTable.updated_at; fix test for non-admin toolset access

* fix: use UniqueViolationError import; guard _ensure_eof for error/cancel only

* fix(mcp): preserve mcp_access_groups in toolset scope, use shared Redis cache for toolset perms

- Remove mcp_access_groups=[] from _apply_toolset_scope (server.py) and the
  responses API toolset path (litellm_proxy_mcp_handler.py). A key's access-group
  grants remain valid even when the request is scoped to a single toolset; clearing
  them silently revoked legitimate entitlements.

- Switch resolve_toolset_tool_permissions and get_toolset_by_name_cached to use
  user_api_key_cache (Redis-backed DualCache in production) instead of per-instance
  in-memory dicts. Cache entries are now shared across workers, eliminating the
  per-worker stale-toolset-permission window flagged as a P1 by Greptile.

- Use union merge (set union of tool names per server) when applying toolset
  permissions in the responses API path so direct-server tool restrictions are not
  overwritten by toolset permissions.

* fix(mcp): return 404 when edit_mcp_toolset target does not exist

* fix(mcp): align mcp_toolsets default to None in LiteLLM_ObjectPermissionTable

* fix(mcp): admin toolset visibility, in-place tool name mutation, test helper coercion

* fix(mcp): treat None/[] team mcp_toolsets as no restriction in key validation

* fix(mcp): allow_all_keys backward compat, blocked_tools API write-path, efficient startup query

* fix(mcp): use _mcp_active_toolset_id ContextVar to detect toolset scope, avoiding DB-default false-positive

* fix(mcp): remove dead toolset cache stubs, log invalidation failures, align schema updated_at defaults

* fix(mcp): deserialise MCPToolset from Redis cache hit, replace fastapi import in test

* fix(mcp): evict name-cache on toolset mutation, 409 on rename conflict, warning-level list errors

* fix(redis): regenerate GCP IAM token per connection for async cluster (#24426)

* fix(redis): regenerate GCP IAM token per connection for async cluster clients

Async RedisCluster was generating the IAM token once at startup and
storing it as a static password. After the 1-hour GCP token TTL, any
new connection (including to newly-discovered cluster nodes) would fail
to authenticate.

Fix: introduce GCPIAMCredentialProvider that implements redis-py's
CredentialProvider protocol. It calls _generate_gcp_iam_access_token()
on every new connection, matching what the sync redis_connect_func
already does. async_redis.RedisCluster accepts a credential_provider
kwarg which is invoked per-connection.

* refactor(redis): move GCPIAMCredentialProvider to its own file

Extract GCPIAMCredentialProvider and _generate_gcp_iam_access_token
into litellm/_redis_credential_provider.py. _redis.py imports them
from there, keeping the public API unchanged.

* fix: address Greptile review issues

- GCPIAMCredentialProvider now inherits from redis.credentials.CredentialProvider
  so redis-py's async path calls get_credentials_async() properly
- move _redis_credential_provider import to top of _redis.py (PEP 8)
- remove dead else-branch that silently no-oped (gcp_service_account from
  redis_kwargs.get() was always None since it's popped by _get_redis_client_logic)
- remove mid-function 'from litellm import get_secret_str' inline import
- remove unused 'call' import from test_redis.py

* chore: retrigger CI/review

* chore: sync schema.prisma copies from root

* chore: sync schema.prisma copies from root

* fix(proxy_server): use bounded asyncio.Queue with maxsize to prevent unbounded growth

* fix(a2a/pydantic_ai): make api_base Optional to match base class signature

* fix(a2a/pydantic_ai): make api_base Optional in handler and guard against None

* fix(mcp): remove unused get_all_mcp_servers import

* fix(mcp): remove unused MCPToolset import

* refactor(mcp): extract toolset permission logic to reduce statement count below PLR0915 limit

* fix(tests): update reload_servers_from_database tests to mock prisma directly

---------

Co-authored-by: Ishaan Jaff <ishaanjaffer0324@gmail.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

* fix(toolset_db): lazy-import prisma to avoid ImportError when prisma not installed

* fix(tests): update UI tests for toolset tab and updated empty state text

* fix(tests): add get_mcp_server_by_name to fake_manager stub

---------

Co-authored-by: Ishaan Jaff <ishaanjaffer0324@gmail.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2026-04-04 16:23:21 -07:00
Ryan Crabbe
c663926fb8
Merge remote-tracking branch 'origin/main' into litellm_feat-add-guardrail 2026-04-04 16:17:38 -07:00