Commit Graph

30881 Commits

Author SHA1 Message Date
Ishaan Jaff
37b6814400
[Refactor] Use Existing config structure for bedrock vector stores (#12672)
* add transform_search_vector_store_request

* add BedrockVectorStoreConfig

* unit tests

* TestBedrockVectorStore
2025-07-17 11:55:11 -07:00
Ishaan Jaff
95f7290e44
[Feat] Proxy - New LLM API Routes /v1/vector_stores and /v1/vector_stores/vs_abc123/search (#12699)
* add Search + Create vector stores

* add new vector_stores routes

* add avector_store_search routes

* docs vector stores

* docs vector stores

* Create Vector Store

* fix - refactor

* add vector_store_router

* fix imports

* fix ruff
2025-07-17 11:19:25 -07:00
Ishaan Jaff
52a525ab9c
[Feat] Bedrock Guardrails - Allow disabling exception on 'BLOCKED' action (#12693)
* flag: disable_exception_on_block

* allow setting disable_exception_on_block

* test_bedrock_guardrail_disable_exception_on_block_non_streaming

* docs Disabling Exceptions on Bedrock BLOCK

* fix recognition_metadata test
2025-07-17 08:46:33 -07:00
Cole McIntosh
77191c7b2a
Fix incorrect environment variable names in LiteLLM gateway documentation (#12686)
Updated ANTHROPIC_API_KEY to ANTHROPIC_AUTH_TOKEN to match the official documentation requirements for LiteLLM gateway configuration.
2025-07-17 08:02:28 -07:00
Cole McIntosh
3a0ea80c75
Add Claude Code LiteLLM tutorial (#12650)
* Add concise Claude Code + LiteLLM Gateway tutorial

- Create focused tutorial matching existing tutorial style
- Step-by-step guide from installation to advanced configurations
- Multi-provider configuration examples (AWS Bedrock, Azure OpenAI, Load Balancing)
- Based on Anthropic's official LiteLLM configuration documentation
- Added to sidebar with clean title 'Use LiteLLM with Claude Code'
- Fixed sidebar reference from 'secret' to 'set_keys' for proper document resolution

* Update config_settings.md to correct documentation links for key management and Hashicorp Vault settings. Changed references from 'secret.md' to 'set_keys.md' for improved clarity and accuracy.

* Update sidebar and config_settings.md to reflect changes in key management documentation. Changed sidebar reference from 'set_keys' to 'secret' and updated links in config_settings.md for Hashicorp Vault settings to point to 'secret.md' for improved accuracy.

* Remove extra tutorial and update sidebar accordingly

* Update tutorial title from 'WebUI' to 'Open WebUI' for clarity and consistency in documentation.

* Remove Python version requirement from Claude Responses API tutorial for clarity and to align with updated prerequisites.
2025-07-16 23:07:29 -07:00
Jugal D. Bhatt
4b09d0d517
[Liveness/Liveliness probe] add separate health app for liveness probes in files (#12669)
* add separate health app

* add new docs

* refactor

* fix colons

* Update config_settings.md

* refactor

* docs

* add unit test
2025-07-16 20:35:09 -07:00
Ishaan Jaff
e5f0a8477b
[Feat] New Vector Store - PG Vector (#12667)
* add PGVectorStoreConfig

* add PGVectorStoreConfig

* test_environment_variable_support

* fix code QA check

* rename test

* add PG vector img

* allow adding vector stores

* add pg vector

* add vector store

* TestPGVectorStoreConfig

* TestPGVectorStoreConfig
2025-07-16 18:17:05 -07:00
Cole McIntosh
b2080ec9af
feat: add input_fidelity parameter for OpenAI image generation (#12662)
* feat: add input_fidelity parameter for OpenAI image generation

- Add input_fidelity to OpenAIImageGenerationOptionalParams type
- Update image_generation function signature to accept input_fidelity
- Add input_fidelity to default_params in get_optional_params_image_gen
- Include input_fidelity in openai_params list for proper handling
- Update documentation with input_fidelity parameter description
- Add test for input_fidelity parameter functionality

This enables control over how closely the model follows the input prompt
for gpt-image-1 model, improving prompt adherence and image quality.

* feat: add input_fidelity to optional parameters for image generation

- Include input_fidelity in the list of OpenAIImageGenerationOptionalParams
- This addition enhances the flexibility of image generation by allowing control over input fidelity.

* test: enhance test for gpt-image-1 with input_fidelity parameter

- Update test_gpt_image_1_with_input_fidelity to include mocking of OpenAI response
- Validate that the OpenAI client is called with correct parameters, including input_fidelity
- Improve response validation to ensure expected output structure and values
2025-07-16 16:56:05 -07:00
Ishaan Jaff
d9943f9812 fix cohere InternalServerError error mapping 2025-07-16 16:13:34 -07:00
Ishaan Jaff
0141ef924f test_create_mcp_server_invalid_alias 2025-07-16 15:58:34 -07:00
Ishaan Jaff
43e7d5288f
[Bug Fix] SCIM - add GET /ServiceProviderConfig (#12664)
* add scimv2 config endpoint

* test_get_service_provider_config
2025-07-16 15:43:25 -07:00
Ishaan Jaff
6a7aab7b84
[Bug Fix] StandardLoggingPayload on cache_hits should track custom llm provider + DD LLM Obs span type (#12652)
* bug fix - ensure custom llm provider is tracked on cache hit

* fix config.yaml

* test_cache_hit_includes_custom_llm_provider

* fix _get_datadog_span_kind

* test_datadog_span_kind_mapping

* fix ruff check

* test_datadog_span_kind_mapping
2025-07-16 15:43:15 -07:00
Ishaan Jaff
538339e1a8
[Feat] UI - Add end_user filter on UI (#12663)
* add end_user filter on Logs

* add end user filter
2025-07-16 15:20:31 -07:00
Ishaan Jaff
9b817b511e
[Feat] Allow reading custom logger python scripts from s3 (#12623)
* add _load_instance_from_remote_storage

* load from s3

* fix download_python_file_from_s3

* fix download_python_file_from_s3

* add _load_instance_from_remote_storage

* tests

* Step 2b - Loading Custom Callbacks

* docs

* fix code qa check
2025-07-16 15:07:01 -07:00
Jugal D. Bhatt
83b0c4cba7
[MCP Gateway] Allow MCP sse and http to have namespaced url for better segregation LIT-304 (#12658)
* fix tools fetch for keys

* Add namespacing in url

* add test for namespacing url

* helper method

* fix test
2025-07-16 14:47:45 -07:00
Jugal D. Bhatt
a7d0b122b9
[MCP Gateway] List tools from access list for keys (#12657)
* fix tools fetch for keys

* add unit test
2025-07-16 14:28:53 -07:00
Jugal D. Bhatt
b3a214bce7
[MCP Gateway] Allow MCP access groups to be added via the config LIT-312 (#12654)
* allow mcp access groups to be added via the config

* fix code and doc quality

* fix mypy

* create helpers:
2025-07-16 13:57:43 -07:00
Krrish Dholakia
6e426c8b7c refactor(admin_ui_sso.md): refactor to include /sso/debug/callback in this doc
more natural place
2025-07-16 11:39:02 -07:00
Krrish Dholakia
047b106197 docs(custom_auth.md): add virtual keys doc 2025-07-16 11:09:57 -07:00
Krrish Dholakia
0fb7c01382 fix(make_model_public_form.tsx): make distinction clearer 2025-07-16 10:58:23 -07:00
Krrish Dholakia
8a1a90bc7a test: update test 2025-07-16 10:28:08 -07:00
Cole McIntosh
9c4b4303d3
fix: remove unused imports in completion_extras transformation (#12655)
- Remove unused GenericResponseOutputItem import
- Remove unused OutputFunctionToolCall import
- Fixes F401 linting errors
2025-07-16 10:23:13 -07:00
Krrish Dholakia
aa12a868c5 fix(migrate_keys.py): add script for migrating keys to new db 2025-07-16 10:18:36 -07:00
Cole McIntosh
d51aee1b84
Add GitHub Copilot LiteLLM tutorial (#12649)
* Add comprehensive GitHub Copilot + LiteLLM integration tutorial

- Complete setup guide from installation to production deployment
- Multiple configuration examples including authentication, load balancing, and cost tracking
- Docker and Kubernetes deployment configurations
- Troubleshooting section with common issues and solutions
- Best practices for security, monitoring, and reliability
- Usage examples for code completion, chat interface, and direct API integration

* Add concise GitHub Copilot + LiteLLM tutorial

- Create focused tutorial matching Gemini CLI style
- Step-by-step guide from installation to production deployment
- Multi-provider configuration examples (OpenAI, Anthropic, Bedrock)
- Load balancing and fallback configuration
- Docker deployment instructions
- Troubleshooting section with common issues
- Updated sidebar with clean title 'Use LiteLLM with GitHub Copilot'

* Refactor GitHub Copilot integration tutorial

- Removed outdated production deployment and direct API usage sections
- Streamlined troubleshooting steps for clarity
- Ensured documentation aligns with current best practices and configurations

* Add proper credit to Sergio Pino for GitHub Copilot tutorial

- Reference original DEV.to article in info box
- Add credits section acknowledging foundational work
- Maintain attribution to original author's guide
2025-07-16 09:40:27 -07:00
Krrish Dholakia
f4131b023e fix: don't fail request if unmapped item in responses list
not every responses item has a 1:1 mapping with chat completions
2025-07-16 09:25:26 -07:00
Ishaan Jaff
e8a748161f
[Bug Fix] grok-4 does not support the stop param (#12646)
* bug fix - using stop reason with grok 4

* fixes for XAI stop params

* test_xai_grok_4_stop_not_supported

* test_xai_grok_4_stop_not_supported
2025-07-16 09:19:25 -07:00
Krrish Dholakia
604075a36c test: update test 2025-07-16 09:15:05 -07:00
Krrish Dholakia
9cac629ca6 test: update test 2025-07-16 09:13:16 -07:00
Krrish Dholakia
446ed6039e docs(admin_ui_sso.md): document /fallback/login flow 2025-07-16 09:07:42 -07:00
Krrish Dholakia
5a8762b6a1 test: update tests 2025-07-16 08:56:41 -07:00
Cole McIntosh
c0935b9d58
Merge pull request #12648 from colesmcintosh/add-groq-moonshotai-kimi-k2-instruct
Add groq/moonshotai-kimi-k2-instruct model configuration
2025-07-16 09:50:52 -06:00
Cole McIntosh
ca40efab31 feat: add groq/moonshotai-kimi-k2-instruct model configuration
- Add model configuration for groq/moonshotai-kimi-k2-instruct
- Set max_tokens: 131072, max_input_tokens: 131072, max_output_tokens: 16384
- Configure pricing: 1e-06 input cost, 3e-06 output cost per token
- Enable function calling, response schema, reasoning, and tool choice support
2025-07-16 09:22:42 -06:00
Ishaan Jaff
0d1a1cfcfb
add together_ai/moonshotai/Kimi-K2-Instruct (#12645) 2025-07-16 07:53:27 -07:00
Michael Nguyen
f505977b62
Update bedrock nova micro and lite info (#12619) 2025-07-16 07:53:15 -07:00
Cole McIntosh
79e4d77bcf
fix: Handle circular references in spend tracking metadata JSON serialization (#12643)
* fix: Handle circular references in spend tracking metadata JSON serialization

- Fixes issue #12634 where circular references in metadata caused
  ValueError: Circular reference detected when logging spend data
- Adds _safe_json_dumps() function that detects and handles circular
  references by replacing them with placeholder strings
- Maintains full functionality for normal objects while preventing
  crashes from circular references
- Adds comprehensive tests for circular reference handling
- Critical fix for v1.74.3 stable release

* fix: Replace bare except clauses with specific Exception handling

- Fixes E722 linting errors in _safe_json_dumps function
- Maintains same error handling behavior while following best practices
- All tests continue to pass

* refactor: Use existing safe_dumps utility instead of custom implementation

- Replace custom _safe_json_dumps() with existing safe_dumps() from litellm_core_utils
- Remove duplicate code and leverage existing circular reference handling
- Update tests to use safe_dumps function
- Maintains same functionality while reducing code duplication
- All tests continue to pass
2025-07-16 07:24:41 -07:00
Krrish Dholakia
4e9440ee85 docs: cleanup docs 2025-07-16 07:18:13 -07:00
Krrish Dholakia
e22390a39a docs(openai.md): cleanup bridge doc 2025-07-15 22:59:10 -07:00
Krrish Dholakia
7064542504 docs(openai.md): document openai chat completions to responses api bridge 2025-07-15 22:50:23 -07:00
Krish Dholakia
1ce3558f96
fix(transformation.py): allows passing native responses api tools like web_search_preview, and mcp via .completion() (#12627)
Closes https://github.com/BerriAI/litellm/issues/12105
2025-07-15 22:45:42 -07:00
Krish Dholakia
955b504f7b
fix(proxy_server.py): fixes for handling team only models via `/v2/mo… (#12632)
* fix(proxy_server.py): fixes for handling team only models via `/v2/model/info`

ensures team only models show up on the correct team on `Models + Endpoints`

* test: update tests
2025-07-15 22:36:05 -07:00
Krish Dholakia
3ad4d9fc3e
fix(router.py): use more descriptive error message (#12629)
* fix(router.py): use more descriptive error message

* fix(proxy/_types.py): note `/team/member_update` is a self-managed route

route has it's own logic for rbac - enables team admins to update member permissions

Fixes issue where team admins on UI could not update member permissions

* fix(token_counter.py): move log line to being '.debug' instead of '.error'

Fixes https://github.com/BerriAI/litellm/issues/12269
2025-07-15 22:34:20 -07:00
Krrish Dholakia
85fe1d35e1 test: update test, remove old gemini models 2025-07-15 22:31:49 -07:00
Ishaan Jaff
84261f3ac8 test_create_delete_assistants 2025-07-15 21:35:25 -07:00
Ishaan Jaff
d8327b4740
[Bug Fix] [Bug]: Knowledge Base Call returning error (#12628)
* bug fix using vector stores as tools

* test_e2e_bedrock_knowledgebase_retrieval_with_llm_api_call_with_tools
2025-07-15 21:33:49 -07:00
yeahyung
782969bb05
(#11794) use upsert for managed object table rather than create to avoid UniqueViolationError (#11795)
* (#11794) use upsert for managed object table rather than create to avoid UniqueViolationError

* (#11794) use upsert for managed object table rather than create to avoid UniqueViolationError
2025-07-15 20:20:01 -07:00
Richard Tweed
197e7efa8f
fix: role chaining with webauthentication for aws bedrock (#12607)
* fix(bedrock): auto-generate session name when only aws_role_name is provided

Fixes #12583 - AWS role assumption not working correctly when aws_role_name
is provided without aws_session_name.

Previously, if only aws_role_name was provided in the config without
aws_session_name, the code would fall back to using environment credentials
instead of assuming the specified role. This was problematic in EKS/IRSA
environments where users want to assume a different role.

The fix:
- When aws_role_name is provided without aws_session_name, we now
  auto-generate a session name with format 'litellm-session-{timestamp}'
- This ensures role assumption happens as expected
- Added comprehensive test coverage for this scenario

* style: format test file with black

---------

Co-authored-by: openhands <openhands@all-hands.dev>
2025-07-15 20:17:09 -07:00
Daniele Scasciafratte
760ece5c8b
Update router.py (#12604) 2025-07-15 20:16:17 -07:00
Stefan Candra
685abf6871
Add token pricing for Together.ai Llama-4 and DeepSeek models (#12622)
* feat: add token pricing for Together.ai Llama-4 and DeepSeek models

* fix typo
2025-07-15 20:15:47 -07:00
Marc Abramowitz
40ccd2b70f
Add "keys import" command to CLI (#12620)
* Add "keys import" command to CLI

E.g.:

```
litellm-proxy keys import \
  --source-base-url=https://old-litellm.company.com \
  --source-api-key=$LITELLM_KEY \
  --dry-run
```

* Add --created-since option

* Add tests

* Fix lint errors

* Fix lint issues

* Fix lint errors

* Fix response.raise_for_status not being a thing

* Fix a mypy error
2025-07-15 20:14:43 -07:00
Stuart Geiger
e758b1b65f
rm claude instant 1 and 1.2 from model prices json (#12631) 2025-07-15 20:13:34 -07:00