Go to file

yuneng-jiang 79a5a7abad feat(tests): behavior-pinning harness + Key Tier-1 matrix (#28321 ) * test(proxy_behavior): scaffold session-scoped async ASGI client + liveness smoke Slice 2 of the management-endpoints behavior-pinning effort. New top-level dir tests/proxy_behavior/management/ outside every existing pytest glob. conftest.py initialises the proxy app once per session against the DATABASE_URL the harness boots Postgres at, wraps it in httpx.AsyncClient via in-process ASGITransport. The one smoke test asserts /health/liveliness returns 200, which exercises the full FastAPI middleware stack against a real app — no mocks. Plan: https://www.notion.so/36643b8acdab8128a581ced0f6a4744d * test(proxy_behavior): connect prisma via real lifespan; key/generate de-risk Slice 3 of the management-endpoints behavior-pinning effort. The fixture now enters the real FastAPI lifespan (proxy_startup_event) instead of just calling initialize() — that is where prisma_client is connected, password migration is kicked off, and the rest of the startup wiring runs. Tests pin the loop to the session scope so the AsyncClient created in the session fixture and the prisma connection opened in the lifespan share the same loop as the test bodies. New de-risk smoke: POST /key/generate with the master key returns 200, the returned sk- token resolves to a hashed row in LiteLLM_VerificationToken, and the cleartext token is never stored. Proves auth + handler + helper + prisma all wire together end-to-end against a real Postgres. Plan: https://www.notion.so/36643b8acdab8128a581ced0f6a4744d * test(proxy_behavior): seed 8-actor read-world for the authz matrix Slice 4 of the management-endpoints behavior-pinning effort. New ``actors.py`` defines the actor enum + seeds an immutable world (2 orgs, 2 teams, 8 users, 8 verification tokens) under the ``behavior-pin-`` prefix so the rows are identifiable in psql and ``_wipe_world`` is targeted. Each actor key is created with its cleartext form generated locally and its hashed form (via ``litellm.proxy.utils.hash_token``) stored in ``LiteLLM_VerificationToken`` — so the real ``user_api_key_auth`` accepts the cleartext bearer token. Roles, ``team_id``, ``organization_id``, and the service-account metadata flag are all set on the seeded rows so the auth layer resolves the same scopes a real proxy would. The session-scoped ``world`` fixture re-seeds at session start (idempotent via wipe-then-create), and the smoke test confirms each of the 8 actor keys can call ``/key/info`` on itself and receive its own row back. Plan: https://www.notion.so/36643b8acdab8128a581ced0f6a4744d * test(proxy_behavior): per-test scratch namespace + targeted delete_many teardown Slice 5 of the management-endpoints behavior-pinning effort. Adds the ``scratch`` function-scoped fixture: each test gets a uuid4-derived namespace prefix, tags writes with it (``key_alias``, ``team_alias``, ``user_id``, ``budget_id``), and the fixture teardown ``delete_many``-s any row whose namespace column starts with that prefix. Cleanup uses Prisma model methods only (no raw SQL, per CLAUDE.md) and orders deletes children-before-parents to avoid FK conflicts. The Slice 3 de-risk smoke is migrated onto the same fixture so it stops accumulating untagged tokens across repeated local runs. Smoke proves both halves of the contract: one test writes a scratch-tagged key and asserts it lands; a second test runs after the first's teardown and asserts no rows in the scratch namespace survived. Plan: https://www.notion.so/36643b8acdab8128a581ced0f6a4744d * test(proxy_behavior): codify G3 (strict-import grep) as a pytest item Slice 6 of the management-endpoints behavior-pinning effort. Two new tests walk every .py file under tests/proxy_behavior/ and assert: * no ``from litellm.proxy.management_endpoints`` import — the suite is deliberately constrained to the HTTP boundary so it survives handler refactors; * no ``mock``/``patch`` on ``user_api_key_auth`` — mocking auth is the structural failure mode of the existing 11k-line mock suite, and the point of this harness is that the real auth layer runs. Codifying G3 as a CI test removes the "did someone forget to check the PR-description checklist" failure mode. Plan: https://www.notion.so/36643b8acdab8128a581ced0f6a4744d * style(proxy_behavior): apply black to G3 grep test Follow-up to 6f588c753b — line-length fixes only, no behavior change. * test(proxy_behavior): pin /key/generate authz matrix (18 scenarios) Slice 7 of the management-endpoints behavior-pinning effort. Parametrized matrix across two axes: actor (8 seeded) × target scope (self, team_alpha in org_a, team_beta in org_b). 18 scenarios after dropping non-applicable combos. Whole-suite wall-time stays at ~4.7s (well under the 10-min G2 budget for the eventual CI job). While pinning, the test surfaced one seed gap: ``_get_user_in_team`` reads ``members_with_roles`` (a JSON list of ``{user_id, role}``), not the plain ``members`` String[]. Both columns are now populated in the seed to match what the real ``/team/new`` handler would produce. Expected status codes are intentionally heterogeneous (200, 400, 401) because the current handler emits different statuses depending on which check fails first (role gate, team-member-perm gate, "not assigned" check). Pinning the observed codes — not what they "should" be — is exactly the regression signal we want. Plan: https://www.notion.so/36643b8acdab8128a581ced0f6a4744d * test(proxy_behavior): pin /key/info authz matrix (24 scenarios) Slice 8 of the management-endpoints behavior-pinning effort. 8 actors × 3 target keys (own, OWNER's key in org_a, CROSS_ORG_USER's key in org_b) covering self-read, same-team-peer read, and cross-org read. Notable pinned behaviors (intentionally surfaced for review, not "fixed"): * ORG_ADMIN gets 403 on individual key info even within their own org — visibility is scoped to "your own keys" + "your team's keys", not "your org's keys". * Same-team peers (INTERNAL_USER, UNRELATED_SAME_ORG, SERVICE_ACCOUNT) DO see each other's keys. Whether that is desired is for the team to decide; this PR only pins the existing behavior so unintentional changes flip the matrix red. Wall-time is unchanged (~4.3s for the slice on its own). Plan: https://www.notion.so/36643b8acdab8128a581ced0f6a4744d * test(proxy_behavior): pin /key/list default-visibility matrix (8 scenarios) Slice 9 of the management-endpoints behavior-pinning effort. For /key/list the response IS the matrix: each of the 8 seeded actors calls the endpoint with default filters and the test asserts set-equality between the returned visible-token set (filtered to seeded tokens only, so unrelated rows can't flap the assertion) and a pinned expected actor-set. Pinned default visibility: * PROXY_ADMIN sees all 8 actors' keys. * Every other actor sees only their own key — including ORG_ADMIN (which had broader expectations going in but currently behaves same-as-internal-user for /key/list defaults) and TEAM_ADMIN (no team-aggregation without include_team_keys=true). Future changes that broaden or narrow any single actor's default visibility will turn this matrix red — exactly the regression signal we want. Parameter-driven views (include_team_keys, filters) are deferred to Slice 13 / PR2 follow-up. Plan: https://www.notion.so/36643b8acdab8128a581ced0f6a4744d * test(proxy_behavior): pin /key/update authz matrix + mutation re-read (21 scenarios) Slice 10 of the management-endpoints behavior-pinning effort. 8 actors × 3 target shapes (self-owned, OWNER-scoped in org_a/team_alpha, CROSS_ORG_USER-scoped in org_b/team_beta) = 21 applicable scenarios. Each test: 1. Master-key-seeds a fresh scratch key with the target's (user_id, team_id) scope (so the read-world stays untouched). 2. Has the actor under test POST /key/update flipping ``models`` to a known marker list. 3. Asserts the status code AND the DB row's ``models`` field — present when 200, unchanged otherwise — so a handler that silently mutates on a denied response surfaces red. Observed gating (pinned, not endorsed): * PROXY_ADMIN bypasses every check. * ORG_ADMIN is blocked by an early role gate, always 401. * Every other (INTERNAL_USER-rolesed) actor hits one of three failure modes — 403 "user can only create keys for themselves", 403 "only proxy admins, team admins, or org admins", or 401 "team_member_permission_error" — depending on whether they own the target and whether they're a team admin / member of its team. Plan: https://www.notion.so/36643b8acdab8128a581ced0f6a4744d * test(proxy_behavior): pin /key/regenerate authz matrix + rotation contract (22 scenarios) Slice 11 of the management-endpoints behavior-pinning effort. 21 matrix scenarios (8 actors × 3 target shapes, minus the cross_org/owner combo that exists in the seed but isn't applicable) plus one smoke for the ``/key/{key:path}/regenerate`` route registration. On 200 outcomes the test verifies the full rotation contract: * the regenerate response key differs from the old cleartext, * the OLD cleartext returns 401 on a follow-up ``/key/info``, * the NEW cleartext returns 200 on a follow-up ``/key/info``. On denied outcomes the test verifies the OLD cleartext still works — catching any handler that mutates the token row on a failed call. Pinned authz divergence vs /key/update: regenerate routes most denials through the team-member-perm 401 path rather than the role-gate 403 path. The matrices for both endpoints are now in tree side-by-side, so any future refactor that "harmonises" the codes will turn one of the two red. Plan: https://www.notion.so/36643b8acdab8128a581ced0f6a4744d * test(proxy_behavior): pin /key/delete authz matrix + post-delete contract (21 scenarios) Slice 12 of the management-endpoints behavior-pinning effort. Mirrors slices 10/11. On success: cleartext can no longer authenticate (handles both hard-delete and soft-delete to LiteLLM_DeletedVerificationToken). On denial: row survives and cleartext still authenticates. Notable behavior gap with /key/update: same-team peers (internal_user, unrelated_same_org, etc.) get 403 on /key/delete for OWNER's key — i.e. cannot delete each other's keys — whereas they CAN read each other's keys (Slice 8). Delete is stricter than read. Pinned as-is. Cumulative whole-suite wall-time is 5.9s for all 128 tests on the local runner — well under the 10-min G2 budget for the CI job in Slice 13. Plan: https://www.notion.so/36643b8acdab8128a581ced0f6a4744d * ci(proxy-mgmt-behavior): add PR-triggered workflow for the behavior suite Slice 13 of the management-endpoints behavior-pinning effort. New workflow ``test-unit-proxy-mgmt-behavior.yml`` fires ``on: pull_request`` for the same branch set every other proxy unit-test workflow watches (main, litellm_internal_staging, litellm_oss_branch, litellm_*). It delegates to the existing reusable ``_test-unit-services-base.yml`` with ``enable-postgres: true``, which already provisions a postgres:14 service container and runs ``prisma db push`` against it before pytest collects. ``reruns: 0`` because a behavior-pinning matrix that needs reruns is itself a regression — flakes are signal. ``timeout-minutes: 15`` gives generous headroom over the local 5.9s whole-suite wall-time; the binding G2 budget is 10 min. Plan: https://www.notion.so/36643b8acdab8128a581ced0f6a4744d docs(proxy_behavior): G4 regression-replay table for Key Tier-1 Slice 14 of the management-endpoints behavior-pinning effort. Documents the regression-replay verification methodology + a 12-row table mapping recent fix-PRs touching key_management_endpoints.py to the catching scenarios in the PR1 matrix. One canonical RED→GREEN cycle is captured verbatim — `c7c3df2b02` "extend /key/update admin check to non-budget fields". Under the parent-of-fix code, 6 scenarios in test_key_update.py flip from 200 to 403; under HEAD code, all 21 pass. The handler swap is the only change between the two runs, confirming the matrix catches the behavior shift the fix introduced. The table also calls out 4 genuine coverage gaps deferred to PR2/PR3: 404-on-missing-key, budget-limit counter assertions, /key/regenerate upperbound enforcement, and /key/list filter-param views. Plan: https://www.notion.so/36643b8acdab8128a581ced0f6a4744d * chore(mutmut): include the behavior suite in tests_dir + G5 triage stub Slice 15 of the management-endpoints behavior-pinning effort. Appends ``tests/proxy_behavior/management/`` to ``[tool.mutmut].tests_dir`` so the existing mutation-test workflow runs against both the legacy mock suite AND the new behavior suite — the latter is where the regression signal will actually surface. Adds a stub at ``tests/proxy_behavior/management/mutmut_triage/pr1.md`` documenting the G5 triage protocol (zero unreviewed survivors in the 6 Tier-1 handler functions) and a placeholder baseline-metrics table to fill in after the first manually-triggered mutmut run completes — runs take hours and run on a manual cadence, so PR1 ships with the wiring + protocol, not the numbers. The actual baseline is recorded in a follow-up once ``gh workflow run mutation-test.yml`` finishes. The kill rate stays telemetry-only, never a gate. G5 (per-survivor classification) is the binding mutation gate. Plan: https://www.notion.so/36643b8acdab8128a581ced0f6a4744d * docs(proxy_behavior): suite README with local-repro + conventions + gates Slice 16 of the management-endpoints behavior-pinning effort. The README documents: * The same three commands the CI workflow runs locally (BYO-DATABASE_URL, no new tooling). * Suite layout — what each test file covers, which slice it lands. * The asyncio loop_scope convention required for session fixtures (httpx AsyncClient + prisma connection) to share a loop with each test body. * G3 strict-import convention + the test that enforces it. * Read-world vs scratch-world fixture conventions. * Behavior-pinning philosophy: pin observed codes; flag, don't judge. * Where each G1–G5 + PR1.M1–M3 gate's evidence lives. Plan: https://www.notion.so/36643b8acdab8128a581ced0f6a4744d * ci(proxy-mgmt-behavior): drop xdist (workers=0) to fix seed race First run on PR #28321 failed with UniqueViolation on ``behavior-pin-budget`` plus cascading missing-membership FK errors. Both xdist workers entered ``seed_world()`` concurrently against the shared Postgres service container; whichever lost the race left the world in a half-seeded state and downstream tests ran against missing team_membership rows. Whole-suite wall-time is ~7s sequentially, so disabling xdist here costs nothing — and the seed itself is the wrong place to add per-worker isolation (the world is intentionally shared so set-equality assertions in /key/list have a deterministic expected set). * ci(proxy-mgmt-behavior): seed scratch keys via proxy_admin actor, not master Second CI run failed: ``/key/generate`` with explicit ``user_id`` returned 403 "User can only create keys for themselves. Got user_id=X, Your ID=None" in every test that called ``_create_scratch_key`` with a per-actor user_id. The bare master key's auth path was producing ``user_id=None`` in the fresh CI Postgres, which doesn't trigger the PROXY_ADMIN bypass in ``_user_can_only_create_keys_for_themselves`` reliably. Locally the same master key path worked, masking the issue. Fix: every ``_create_scratch_key`` helper now takes a seeder cleartext and the test bodies pass ``world.keys[Actor.PROXY_ADMIN].cleartext``. That actor was seeded with ``user_role=PROXY_ADMIN`` AND a concrete ``user_id``, so the bypass fires deterministically in both environments. No behavior shift in the matrices themselves — all 128 scenarios still pass locally; only the setup helper's auth identity changed. The bare-master smoke (test_smoke + test_scratch_teardown) is intentionally left on the master key path: those tests don't pass ``user_id`` in the body so they don't hit the user_id-mismatch gate. * ci(proxy-mgmt-behavior): diag — run world-seed test first + bump max-failures Third CI run failed identically: seeded PROXY_ADMIN actor's auth resolves to ``user_id=None`` even though the DB row has the right ``user_id``. The suite was aborting at maxfail=10 inside test_key_delete, so test_world_seed (which would tell us whether the seed itself is reachable) never ran in CI. Two diagnostic moves on this push, no behavior change: * Rename ``test_world_seed.py`` → ``test_aaa_world_seed.py`` so it's the first collected file. If it passes in CI we know the seed is fine and the bug lives downstream; if it fails the same way the bug is in the auth resolution path. * Bump ``max-failures`` to 200 for this workflow so we see the full failure surface instead of stopping at the first cascading setup error. Will tighten back down once the suite is green. Adds one new test ``test_proxy_admin_actor_can_create_keys_for_others`` that explicitly exercises the PROXY_ADMIN bypass via /key/generate with an explicit user_id — the same shape the matrix setup helper uses but without the matrix machinery muddying the diagnostic. * ci(proxy-mgmt-behavior): await LiteLLM_VerificationTokenView creation in fixture Fourth CI run still failed because the proxy's lifespan kicks off ``prisma_client.check_view_exists()`` as a fire-and-forget background task — that task is what creates ``LiteLLM_VerificationTokenView``, the SQL view ``user_api_key_auth`` queries to resolve a token to its user_id / user_role / team. On a fresh Postgres (CI), the first test races the background task. The view doesn't exist when the first auth call runs, the resolver falls through to a degraded path that returns ``user_id=None``, and every matrix test that depends on the seeded actor's identity then fails confusingly with "Got user_id=X, Your ID=None" 403s. Locally the view persists across pytest runs so the race is invisible. Fix: await ``prisma_client.check_view_exists()`` explicitly inside the session ``proxy_app`` fixture, after the lifespan enters but before the fixture yields. Deterministic regardless of whether the underlying DB is fresh (CI) or warm (local). * ci(proxy-mgmt-behavior): widen diagnostic to dump token / user / view shape The fifth CI run isolated the failure to ``/key/generate`` with explicit user_id while ``/key/info`` works for the same seeded PROXY_ADMIN actor. The auth context's user_id is None even though the DB row has it set. This commit widens the diagnostic test: on failure, dump the raw token row's user_id, the user row's user_role, and what ``LiteLLM_VerificationTokenView`` actually returns for the seeded token. If the view returns user_id=None we know the view shape is the problem; if the view returns the right user_id we know it's a downstream code path stripping it. * ci(proxy-mgmt-behavior): unambiguous diagnostic view query Previous diagnostic's raw SQL had an ambiguous user_id column from joining the view with the user table, so the diagnostic itself crashed before printing useful state. Simplified to query just the view's columns. * ci(proxy-mgmt-behavior): add auth-resolver chain diagnostic Six runs and the underlying data (token row, user row, view row) all verified correct in CI, but auth still returns user_id=None. This diagnostic calls the resolver primitives directly: 1. ``prisma.get_data(table_name="combined_view")`` → raw view object 2. ``get_key_object(...)`` → cached/DB UserAPIKeyAuth 3. ``get_user_object(...)`` → LiteLLM_UserTable row 4. ``_is_user_proxy_admin`` / ``_get_user_role`` and prints each intermediate via captured stdout (-s). Whichever step returns None/False in CI is where the chain breaks. Imports come from ``litellm.proxy.auth`` (not management_endpoints), so G3 still passes. * ci(proxy-mgmt-behavior): set LITELLM_MASTER_KEY env so lifespan doesn't wipe it Real root cause of every CI run that returned ``Your ID=None`` for the seeded actors: * In ``initialize()``, ``master_key`` is set from the config YAML's ``general_settings.master_key`` (load_config code path at proxy_server.py:4174). * Then the FastAPI lifespan (``proxy_startup_event``) runs and at line 776 does ``master_key = get_secret_str("LITELLM_MASTER_KEY")``, which UNCONDITIONALLY overwrites the global. * In CI the env var is unset, so the post-lifespan ``master_key`` is None. Downstream every auth path degrades: master-key requests don't bypass because ``secrets.compare_digest(api_key, None)`` raises and is caught to ``is_master_key_valid=False``; seeded-actor requests cache a ``UserAPIKeyAuth`` whose ``user_role`` never resolves through the PROXY_ADMIN bypass; ``_is_allowed_to_make_key_request`` then hits the ``user_id`` mismatch path with ``Your ID=None``. Locally my shell happened to have ``LITELLM_MASTER_KEY`` set from a prior session, which is why every local run was green and CI red — exactly the "don't generalize from your environment to CI" memory. Fix: ``os.environ.setdefault("LITELLM_MASTER_KEY", MASTER_KEY)`` and ``os.environ.setdefault("CONFIG_FILE_PATH", config_path)`` before entering the lifespan, so its re-read produces the same value as ``initialize()``. Whole-suite still green locally (130 tests, ~6.4s). * ci(proxy-mgmt-behavior): force premium_user=True so /key/regenerate isn't gated Ninth CI run cleared every ``Your ID=None`` failure (the master_key env fix worked end-to-end) and exposed the next thin layer of failures: ``/key/regenerate`` returns 500 "Regenerating Virtual Keys is an Enterprise feature" in CI because the proxy can't see a ``LITELLM_LICENSE``. Locally my license is set, so the matrix passes. The behavior matrix is supposed to pin authz, not licensing — so flip ``proxy_server.premium_user = True`` directly, both before and after the lifespan (the lifespan re-runs ``_license_check.is_premium()`` and would otherwise reset it). With premium gating disabled, the regenerate matrix exercises the same authz path /key/update does. Whole-suite still green locally (130 tests, ~6.3s). * test(proxy_behavior): trim debug diagnostics, restore default max-failures Followup to the CI-bring-up sequence: now that the suite is green in CI (130 → 129 tests after this trim; 156s wall-time on ubuntu-latest), drop the diagnostic noise left over from debugging the master_key wipe: * Rename ``test_aaa_world_seed.py`` back to ``test_world_seed.py`` — no longer needs to run first. * Remove ``test_auth_resolver_returns_correct_user_id_and_role`` — that test reached into private auth helpers to localize the bug between the DB and ``UserAPIKeyAuth``; it has served its purpose and isn't HTTP-boundary. * Keep ``test_proxy_admin_actor_can_create_keys_for_others`` (without the failure-time dump) — it's a real authz contract that pins the PROXY_ADMIN bypass on /key/generate, and would catch a regression of the same conftest interaction this sequence revealed. * Drop the workflow's ``max-failures: 200`` override — that was a debug aid for seeing the full failure surface in CI. Default of 10 is right for a stable suite. * chore(proxy_behavior): drop empty mutmut triage stub, fold protocol into README The mutmut_triage/pr1.md file was a placeholder for numbers and classifications that don't exist yet — the first mutmut run is a manual follow-up. Empty stubs aren't evidence; deleting it. The G5 protocol (run the workflow, triage survivors in the six Tier-1 handler functions, kill-or-accept-with-reason, zero unreviewed) moves into the suite README's "Gate evidence" block. The real triage file will land alongside the first mutmut follow-up. pyproject.toml's [tool.mutmut].tests_dir entry stays — that's the one-line wiring that makes the existing (manual-trigger) mutation-test workflow include our suite next time someone runs it. Comment updated to drop the dead file reference. * chore(proxy_behavior): drop README + trim comments Removes the suite README — its contents (local repro, layout, conventions) were either restated by the file structure or already covered by the workflow YAML and pyproject.toml. Trims docstrings and inline comments across every test file to keep only non-obvious WHY (the masking ``_get_user_in_team`` reads, the LiteLLM_VerificationTokenView models-can't- be-NULL gotcha, the org_admin/peer-visibility surprise, the rotation contract). Suite still 129 green locally. * test(proxy_behavior): address Greptile review — env force, pagination, dedup - conftest: force LITELLM_MASTER_KEY / CONFIG_FILE_PATH unconditionally instead of setdefault. An ambient LITELLM_MASTER_KEY with a different value would make the proxy authenticate on that key while the tests still send MASTER_KEY → silent 401s. - test_key_list: paginate /key/list instead of a single size=100 request. size is capped at 100 by the endpoint, so on a non-fresh DB a single page could truncate PROXY_ADMIN's view and a seeded key could fall off the page. Walk total_pages. - conftest: hoist the duplicated _create_scratch_key helper (copy-pasted and already diverged across test_key_{update,regenerate,delete}.py) into a single shared create_scratch_key. - Delete regression_replay/README.md — G4 regression-replay evidence belongs in the PR description, not a committed doc file (repo docs policy + the effort's own plan both say so). Content moved to the PR.		2026-05-20 19:27:44 -07:00
.circleci	fix(interactions): never drop streamed text deltas; always emit terminal completion (#28394 )	2026-05-20 16:41:40 -07:00
.devcontainer	build: migrate packaging, CI, and Docker from Poetry to uv (#25007 )	2026-04-09 11:46:23 -07:00
.github	feat(tests): behavior-pinning harness + Key Tier-1 matrix (#28321 )	2026-05-20 19:27:44 -07:00
.semgrep/rules	security: remove .claude/settings.json and add semgrep rule to prevent re-adding	2026-03-25 11:57:43 -07:00
backend	feat: add componentized proxy deployment with gateway, backend, ui, and migrations (#27557 )	2026-05-16 09:25:17 -07:00
ci_cd	Drop dep bumps + black-26 reformat to clear fork CI policy	2026-05-07 23:04:52 +00:00
cookbook	Drop dep bumps + black-26 reformat to clear fork CI policy	2026-05-07 23:04:52 +00:00
db_scripts	Drop dep bumps + black-26 reformat to clear fork CI policy	2026-05-07 23:04:52 +00:00
deploy	Litellm oss staging (#28161 )	2026-05-18 16:27:44 -07:00
dist
docker	chore: remove legacy deployment artifacts and litellm-js packages (#27541 )	2026-05-09 20:51:34 +00:00
docs	fix(hosted_vllm): normalize custom tools for chat completions (#25763 )	2026-05-05 17:27:02 -07:00
enterprise	[Infra] Bump versions (#28094 )	2026-05-16 18:31:43 -07:00
gateway	feat: add componentized proxy deployment with gateway, backend, ui, and migrations (#27557 )	2026-05-16 09:25:17 -07:00
helm/litellm	feat: add componentized proxy deployment with gateway, backend, ui, and migrations (#27557 )	2026-05-16 09:25:17 -07:00
litellm	chore(ci): merge dev branch (#28314 )	2026-05-20 17:47:33 -07:00
litellm-proxy-extras	chore(ci): bump versions (#28287 )	2026-05-19 15:10:37 -07:00
migrations	feat: add componentized proxy deployment with gateway, backend, ui, and migrations (#27557 )	2026-05-16 09:25:17 -07:00
scripts	perf: eliminate per-request callback scanning on proxy hot path (#27858 )	2026-05-14 09:28:31 -07:00
terraform/litellm	feat: add Terraform stacks for deploying LiteLLM on AWS and GCP (#27673 )	2026-05-16 17:26:20 -07:00
tests	feat(tests): behavior-pinning harness + Key Tier-1 matrix (#28321 )	2026-05-20 19:27:44 -07:00
ui	Litellm oss staging 1 (#28337 )	2026-05-20 17:27:03 -07:00
.dockerignore	fix critical CVE vulnerabliltes (#20683 )	2026-02-07 22:23:01 -08:00
.env.example
.flake8
.git-blame-ignore-revs
.gitattributes
.gitguardian.yaml	build: migrate packaging, CI, and Docker from Poetry to uv (#25007 )	2026-04-09 11:46:23 -07:00
.gitignore	feat: add Terraform stacks for deploying LiteLLM on AWS and GCP (#27673 )	2026-05-16 17:26:20 -07:00
.npmrc	[Fix] CI/Tooling: Correct min-release-age value in .npmrc files	2026-04-29 19:49:27 -07:00
AGENTS.md	chore: remove legacy deployment artifacts and litellm-js packages (#27541 )	2026-05-09 20:51:34 +00:00
ARCHITECTURE.md	[Docs] Litellm architecture fixes 2 (#19252 )	2026-01-16 14:52:16 -08:00
CLAUDE.md	chore(mcp): warn on internal + upstream PKCE delegate	2026-05-15 10:05:35 +05:30
codecov.yaml	fix(ci): flag codecov uploads, enable carryforward, close coverage gaps (#28028 )	2026-05-16 10:56:32 -07:00
CONTRIBUTING.md	build: migrate packaging, CI, and Docker from Poetry to uv (#25007 )	2026-04-09 11:46:23 -07:00
cosign.pub	[Infra] Add release workflow and cosign public key	2026-03-31 14:30:27 -07:00
docker-compose.hardened.yml
docker-compose.yml	feat: add read-replica routing for Prisma DB via DATABASE_URL_READ_REPLICA (#27493 )	2026-05-08 21:05:50 -07:00
Dockerfile	fix(docker): restore npm@11.14.0 lost in merge resolution	2026-05-07 17:25:10 -07:00
GEMINI.md	build: migrate packaging, CI, and Docker from Poetry to uv (#25007 )	2026-04-09 11:46:23 -07:00
LICENSE
license_cache.json	feat: add componentized proxy deployment with gateway, backend, ui, and migrations (#27557 )	2026-05-16 09:25:17 -07:00
Makefile	tests(vcr): trim non-load-bearing comments and docstrings	2026-04-30 21:48:48 +00:00
mcp_servers.json
model_prices_and_context_window.json	Litellm oss staging 1 (#28337 )	2026-05-20 17:27:03 -07:00
package-lock.json	chore(deps): refresh dependency locks	2026-05-04 11:36:18 -07:00
package.json	chore(deps): refresh dependency locks	2026-05-04 11:36:18 -07:00
policy_templates.json	feat: Add Canadian PII protection (PIPEDA) (#22951 )	2026-03-06 18:27:31 -08:00
prometheus.yml
provider_endpoints_support.json	feat(audio_transcription): add NVIDIA Riva STT provider (#27185 )	2026-05-05 17:17:51 -07:00
proxy_server_config.yaml	chore(ci): modernize model references in tests and configs (#27856 )	2026-05-15 15:44:28 -07:00
pyproject.toml	feat(tests): behavior-pinning harness + Key Tier-1 matrix (#28321 )	2026-05-20 19:27:44 -07:00
pyrightconfig.json
README.md	Litellm oss staging (#28161 )	2026-05-18 16:27:44 -07:00
render.yaml
ruff.toml	[Fix] CI: fix 6 more CircleCI job failures from uv migration	2026-04-10 21:06:25 -07:00
schema.prisma	Litellm oss staging (#28161 )	2026-05-18 16:27:44 -07:00
security.md	chore: update security.md (#24871 )	2026-03-31 13:13:18 -07:00
taplo.toml	fix(agentcore): simplify agentcore streaming (#17141 )	2026-01-19 05:20:24 -08:00
uv.lock	chore(ci): bump versions (#28287 )	2026-05-19 15:10:37 -07:00

README.md

🚅 LiteLLM

LiteLLM AI Gateway

Open Source AI Gateway for 100+ LLMs. Self-hosted. Enterprise-ready. Call any LLM in OpenAI format.

LiteLLM Proxy Server (AI Gateway) | Hosted Proxy | Enterprise Tier | Website

What is LiteLLM

LiteLLM is an open source AI Gateway that gives you a single, unified interface to call 100+ LLM providers — OpenAI, Anthropic, Gemini, Bedrock, Azure, and more — using the OpenAI format.

Use it as a Python SDK for direct library integration, or deploy the AI Gateway (Proxy Server) as a centralized service for your team or organization.

Jump to LiteLLM Proxy (LLM Gateway) Docs
Jump to Supported LLM Providers

Why LiteLLM

Managing LLM calls across providers gets complicated fast — different SDKs, auth patterns, request formats, and error types for every model. LiteLLM removes that friction:

Unified API — one interface for 100+ LLMs, no provider-specific SDK juggling
Drop-in OpenAI compatibility — swap providers without rewriting your code
Production-ready gateway — virtual keys, spend tracking, guardrails, load balancing, and an admin dashboard out of the box
8ms P95 latency at 1k RPS (benchmarks)

OSS Adopters

Netflix

Features

LLMs - Call 100+ LLMs (Python SDK + AI Gateway)

All Supported Endpoints - /chat/completions, /responses, /embeddings, /images, /audio, /batches, /rerank, /a2a, /messages and more.

Python SDK

uv add litellm

from litellm import completion
import os

os.environ["OPENAI_API_KEY"] = "your-openai-key"
os.environ["ANTHROPIC_API_KEY"] = "your-anthropic-key"

# OpenAI
response = completion(model="openai/gpt-4o", messages=[{"role": "user", "content": "Hello!"}])

# Anthropic  
response = completion(model="anthropic/claude-sonnet-4-20250514", messages=[{"role": "user", "content": "Hello!"}])

AI Gateway (Proxy Server)

Getting Started - E2E Tutorial - Setup virtual keys, make your first request

uv tool install 'litellm[proxy]'
litellm --model gpt-4o

import openai

client = openai.OpenAI(api_key="anything", base_url="http://0.0.0.0:4000")
response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Hello!"}]
)

Docs: LLM Providers

Agents - Invoke A2A Agents (Python SDK + AI Gateway)

Supported Providers - LangGraph, Vertex AI Agent Engine, Azure AI Foundry, Bedrock AgentCore, Pydantic AI

Python SDK - A2A Protocol

from litellm.a2a_protocol import A2AClient
from a2a.types import SendMessageRequest, MessageSendParams
from uuid import uuid4

client = A2AClient(base_url="http://localhost:10001")

request = SendMessageRequest(
    id=str(uuid4()),
    params=MessageSendParams(
        message={
            "role": "user",
            "parts": [{"kind": "text", "text": "Hello!"}],
            "messageId": uuid4().hex,
        }
    )
)
response = await client.send_message(request)

AI Gateway (Proxy Server)

Step 1. Add your Agent to the AI Gateway

Step 2. Call Agent via A2A SDK

from a2a.client import A2ACardResolver, A2AClient
from a2a.types import MessageSendParams, SendMessageRequest
from uuid import uuid4
import httpx

base_url = "http://localhost:4000/a2a/my-agent"  # LiteLLM proxy + agent name
headers = {"Authorization": "Bearer sk-1234"}    # LiteLLM Virtual Key

async with httpx.AsyncClient(headers=headers) as httpx_client:
    resolver = A2ACardResolver(httpx_client=httpx_client, base_url=base_url)
    agent_card = await resolver.get_agent_card()
    client = A2AClient(httpx_client=httpx_client, agent_card=agent_card)

    request = SendMessageRequest(
        id=str(uuid4()),
        params=MessageSendParams(
            message={
                "role": "user",
                "parts": [{"kind": "text", "text": "Hello!"}],
                "messageId": uuid4().hex,
            }
        )
    )
    response = await client.send_message(request)

Docs: A2A Agent Gateway

MCP Tools - Connect MCP servers to any LLM (Python SDK + AI Gateway)

Python SDK - MCP Bridge

from mcp import ClientSession, StdioServerParameters
from mcp.client.stdio import stdio_client
from litellm import experimental_mcp_client
import litellm

server_params = StdioServerParameters(command="python", args=["mcp_server.py"])

async with stdio_client(server_params) as (read, write):
    async with ClientSession(read, write) as session:
        await session.initialize()

        # Load MCP tools in OpenAI format
        tools = await experimental_mcp_client.load_mcp_tools(session=session, format="openai")

        # Use with any LiteLLM model
        response = await litellm.acompletion(
            model="gpt-4o",
            messages=[{"role": "user", "content": "What's 3 + 5?"}],
            tools=tools
        )

AI Gateway - MCP Gateway

Step 1. Add your MCP Server to the AI Gateway

Step 2. Call MCP tools via /chat/completions

curl -X POST 'http://0.0.0.0:4000/v1/chat/completions' \
  -H 'Authorization: Bearer sk-1234' \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "gpt-4o",
    "messages": [{"role": "user", "content": "Summarize the latest open PR"}],
    "tools": [{
      "type": "mcp",
      "server_url": "litellm_proxy/mcp/github",
      "server_label": "github_mcp",
      "require_approval": "never"
    }]
  }'

Use with Cursor IDE

{
  "mcpServers": {
    "LiteLLM": {
      "url": "http://localhost:4000/mcp/",
      "headers": {
        "x-litellm-api-key": "Bearer sk-1234"
      }
    }
  }
}

Docs: MCP Gateway

Supported Providers (Website Supported Models | Docs)

Provider	`/chat/completions`	`/messages`	`/responses`	`/embeddings`	`/image/generations`	`/audio/transcriptions`	`/audio/speech`	`/moderations`	`/batches`	`/rerank`
Abliteration (`abliteration`)	✅
AI/ML API (`aiml`)	✅	✅	✅	✅	✅
AI21 (`ai21`)	✅	✅	✅
AI21 Chat (`ai21_chat`)	✅	✅	✅
Aleph Alpha	✅	✅	✅
Amazon Nova	✅	✅	✅
Anthropic (`anthropic`)	✅	✅	✅						✅
Anthropic Text (`anthropic_text`)	✅	✅	✅						✅
Anyscale	✅	✅	✅
AssemblyAI (`assemblyai`)	✅	✅	✅			✅
Auto Router (`auto_router`)	✅	✅	✅
AWS - Bedrock (`bedrock`)	✅	✅	✅	✅						✅
AWS - Sagemaker (`sagemaker`)	✅	✅	✅	✅
Azure (`azure`)	✅	✅	✅	✅	✅	✅	✅	✅	✅
Azure AI (`azure_ai`)	✅	✅	✅	✅	✅	✅	✅	✅	✅
Azure Text (`azure_text`)	✅	✅	✅			✅	✅	✅	✅
Baseten (`baseten`)	✅	✅	✅
Bytez (`bytez`)	✅	✅	✅
Cerebras (`cerebras`)	✅	✅	✅
Clarifai (`clarifai`)	✅	✅	✅
Cloudflare AI Workers (`cloudflare`)	✅	✅	✅
Codestral (`codestral`)	✅	✅	✅
Cohere (`cohere`)	✅	✅	✅	✅						✅
Cohere Chat (`cohere_chat`)	✅	✅	✅
CometAPI (`cometapi`)	✅	✅	✅	✅
CompactifAI (`compactifai`)	✅	✅	✅
Custom (`custom`)	✅	✅	✅
Custom OpenAI (`custom_openai`)	✅	✅	✅			✅	✅	✅	✅
Dashscope (`dashscope`)	✅	✅	✅	✅						✅
Databricks (`databricks`)	✅	✅	✅
DataRobot (`datarobot`)	✅	✅	✅
Deepgram (`deepgram`)	✅	✅	✅			✅
DeepInfra (`deepinfra`)	✅	✅	✅
Deepseek (`deepseek`)	✅	✅	✅
ElevenLabs (`elevenlabs`)	✅	✅	✅			✅	✅
Empower (`empower`)	✅	✅	✅
Fal AI (`fal_ai`)	✅	✅	✅		✅
Featherless AI (`featherless_ai`)	✅	✅	✅
Fireworks AI (`fireworks_ai`)	✅	✅	✅
FriendliAI (`friendliai`)	✅	✅	✅
Galadriel (`galadriel`)	✅	✅	✅
GitHub Copilot (`github_copilot`)	✅	✅	✅	✅
GitHub Models (`github`)	✅	✅	✅
Google - PaLM	✅	✅	✅
Google - Vertex AI (`vertex_ai`)	✅	✅	✅	✅	✅
Google AI Studio - Gemini (`gemini`)	✅	✅	✅
GradientAI (`gradient_ai`)	✅	✅	✅
Groq AI (`groq`)	✅	✅	✅
Heroku (`heroku`)	✅	✅	✅
Hosted VLLM (`hosted_vllm`)	✅	✅	✅
Huggingface (`huggingface`)	✅	✅	✅	✅						✅
Hyperbolic (`hyperbolic`)	✅	✅	✅
IBM - Watsonx.ai (`watsonx`)	✅	✅	✅	✅
Infinity (`infinity`)				✅
Jina AI (`jina_ai`)				✅
Lambda AI (`lambda_ai`)	✅	✅	✅
Lemonade (`lemonade`)	✅	✅	✅
LiteLLM Proxy (`litellm_proxy`)	✅	✅	✅	✅	✅
Llamafile (`llamafile`)	✅	✅	✅
LM Studio (`lm_studio`)	✅	✅	✅
Maritalk (`maritalk`)	✅	✅	✅
Meta - Llama API (`meta_llama`)	✅	✅	✅
Mistral AI API (`mistral`)	✅	✅	✅	✅
Moonshot (`moonshot`)	✅	✅	✅
Morph (`morph`)	✅	✅	✅
Nebius AI Studio (`nebius`)	✅	✅	✅	✅
NLP Cloud (`nlp_cloud`)	✅	✅	✅
Novita AI (`novita`)	✅	✅	✅
Nscale (`nscale`)	✅	✅	✅
Nvidia NIM (`nvidia_nim`)	✅	✅	✅
OCI (`oci`)	✅	✅	✅
Ollama (`ollama`)	✅	✅	✅	✅
Ollama Chat (`ollama_chat`)	✅	✅	✅
Oobabooga (`oobabooga`)	✅	✅	✅			✅	✅	✅	✅
OpenAI (`openai`)	✅	✅	✅	✅	✅	✅	✅	✅	✅
OpenAI-like (`openai_like`)				✅
OpenRouter (`openrouter`)	✅	✅	✅
OVHCloud AI Endpoints (`ovhcloud`)	✅	✅	✅
Perplexity AI (`perplexity`)	✅	✅	✅
Petals (`petals`)	✅	✅	✅
Predibase (`predibase`)	✅	✅	✅
Recraft (`recraft`)					✅
Replicate (`replicate`)	✅	✅	✅
Sagemaker Chat (`sagemaker_chat`)	✅	✅	✅
Sambanova (`sambanova`)	✅	✅	✅
Snowflake (`snowflake`)	✅	✅	✅
Text Completion Codestral (`text-completion-codestral`)	✅	✅	✅
Text Completion OpenAI (`text-completion-openai`)	✅	✅	✅			✅	✅	✅	✅
Together AI (`together_ai`)	✅	✅	✅
Topaz (`topaz`)	✅	✅	✅
Triton (`triton`)	✅	✅	✅
V0 (`v0`)	✅	✅	✅
Vercel AI Gateway (`vercel_ai_gateway`)	✅	✅	✅
VLLM (`vllm`)	✅	✅	✅
Volcengine (`volcengine`)	✅	✅	✅
Voyage AI (`voyage`)				✅
WandB Inference (`wandb`)	✅	✅	✅
Watsonx Text (`watsonx_text`)	✅	✅	✅
xAI (`xai`)	✅	✅	✅
Xinference (`xinference`)				✅

Read the Docs

Get Started

You can use LiteLLM through either the Proxy Server or Python SDK. Both give you a unified interface to access multiple LLMs (100+ LLMs). Choose the option that best fits your needs:

	LiteLLM AI Gateway	LiteLLM Python SDK
Use Case	Central service (LLM Gateway) to access multiple LLMs	Use LiteLLM directly in your Python code
Who Uses It?	Gen AI Enablement / ML Platform Teams	Developers building LLM projects
Key Features	Centralized API gateway with authentication and authorization, multi-tenant cost tracking and spend management per project/user, per-project customization (logging, guardrails, caching), virtual keys for secure access control, admin dashboard UI for monitoring and management	Direct Python library integration in your codebase, Router with retry/fallback logic across multiple deployments (e.g. Azure/OpenAI) - Router, application-level load balancing and cost tracking, exception handling with OpenAI-compatible errors, observability callbacks (Lunary, MLflow, Langfuse, etc.)

Stable Release: Use docker images with the -stable tag. These have undergone 12 hour load tests, before being published. More information about the release cycle here

Support for more providers. Missing a provider or LLM Platform, raise a feature request.

Run in Developer Mode

Services

Setup .env file in root
Run dependant services docker-compose up db prometheus

Backend

(In root) create virtual environment python -m venv .venv
Activate virtual environment source .venv/bin/activate
Install dependencies uv sync --all-extras --group proxy-dev
uv run prisma generate
prisma generate
Start proxy backend python litellm/proxy/proxy_cli.py

Frontend

Navigate to ui/litellm-dashboard
Install dependencies npm install
Run npm run dev to start the dashboard

Verify Docker Image Signatures

All LiteLLM Docker images published to GHCR are signed with cosign. Every release is signed with the same key introduced in commit 0112e53.

Verify using the pinned commit hash (recommended):

A commit hash is cryptographically immutable, so this is the strongest way to ensure you are using the original signing key:

cosign verify \
  --key https://raw.githubusercontent.com/BerriAI/litellm/0112e53046018d726492c814b3644b7d376029d0/cosign.pub \
  ghcr.io/berriai/litellm:<release-tag>

Verify using a release tag (convenience):

Tags are protected in this repository and resolve to the same key. This option is easier to read but relies on tag protection rules:

cosign verify \
  --key https://raw.githubusercontent.com/BerriAI/litellm/<release-tag>/cosign.pub \
  ghcr.io/berriai/litellm:<release-tag>

Replace <release-tag> with the version you are deploying (e.g. v1.83.0-stable).

Enterprise

For companies that need better security, user management and professional support

Get an Enterprise License Talk to founders

This covers:

✅ Features under the LiteLLM Commercial License:
✅ Feature Prioritization
✅ Custom Integrations
✅ Professional Support - Dedicated discord + slack
✅ Custom SLAs
✅ Secure access with Single Sign-On

Contributing

We welcome contributions to LiteLLM! Whether you're fixing bugs, adding features, or improving documentation, we appreciate your help.

Quick Start for Contributors

This requires uv to be installed.

git clone https://github.com/BerriAI/litellm.git
cd litellm
make install-dev    # Install development dependencies
make format         # Format your code
make lint           # Run all linting checks
make test-unit      # Run unit tests
make format-check   # Check formatting only

For detailed contributing guidelines, see CONTRIBUTING.md.

📖 Contributing to documentation? The LiteLLM docs have moved to a separate repository: BerriAI/litellm-docs. Please open doc PRs there. Docs are served at docs.litellm.ai.

Code Quality / Linting

LiteLLM follows the Google Python Style Guide.

Our automated checks include:

Black for code formatting
Ruff for linting and code quality
MyPy for type checking
Circular import detection
Import safety checks

All these checks must pass before your PR can be merged.

Support / talk with founders

Schedule Demo 👋
Community Discord 💭
Community Slack 💭
Our emails ✉️ ishaan@berri.ai / krrish@berri.ai