Commit Graph

30881 Commits

Author SHA1 Message Date
yujonglee
293bb4c522
fix (#7769) 2025-01-14 14:31:44 -08:00
Ishaan Jaff
8c016d0184 docs benchmark 2025-01-14 10:48:43 -08:00
Ishaan Jaff
eb2770fee2 update benchmarks 2025-01-14 10:45:28 -08:00
Ishaan Jaff
00f50bc201 bump: version 1.58.0 → 1.58.1 2025-01-13 20:20:00 -08:00
Ishaan Jaff
d510f1d517
(fix) health check - allow setting health_check_model (#7752)
* use _update_litellm_params_for_health_check

* fix Wildcard Routes

* test_update_litellm_params_for_health_check

* test_perform_health_check_with_health_check_model

* fix doc string

* huggingface/mistralai/Mistral-7B-Instruct-v0.3
2025-01-13 20:16:44 -08:00
Ishaan Jaff
9daa6fb0b4
(prometheus - minor bug fix) - litellm_llm_api_time_to_first_token_metric not populating for bedrock models (#7740)
* fix prometheus ttft

* fix test_set_latency_metrics

* fix _set_latency_metrics

* fix _set_latency_metrics

* fix test_set_latency_metrics

* test_async_log_success_event

* huggingface/mistralai/Mistral-7B-Instruct-v0.3
2025-01-13 20:16:34 -08:00
Ishaan Jaff
d88f01d518
(litellm SDK perf improvements) - handle cases when unable to lookup model in model cost map (#7750)
* use lru cache wrapper

* use lru_cache_wrapper for _cached_get_model_info_helper

* fix _get_traceback_str_for_error

* huggingface/mistralai/Mistral-7B-Instruct-v0.3
2025-01-13 19:58:46 -08:00
Ishaan Jaff
c8ac61f117
fix http parsing utils (#7753) 2025-01-13 19:58:26 -08:00
Ishaan Jaff
f1335362cf
(core sdk fix) - fix fallbacks stuck in infinite loop (#7751)
* test_acompletion_fallbacks_basic

* use common run_async_function

* fix completion_with_fallbacks

* fix completion with fallbacks

* fix fallback utils

* test_acompletion_fallbacks_basic

* test_completion_fallbacks_sync

* huggingface/mistralai/Mistral-7B-Instruct-v0.3
2025-01-13 19:34:34 -08:00
Ishaan Jaff
970e9c7507 huggingface/mistralai/Mistral-7B-Instruct-v0.3 2025-01-13 18:42:36 -08:00
ProphetJeremy
ecf6d22dc2
(docs) Update vertex.md old code example
Complete imports
Remove invalid parameter `disable_atributon`
2025-01-13 14:27:27 +01:00
Ishaan Jaff
3fe1f3b3b2 test_team_access_groups 2025-01-12 22:26:13 -08:00
Ishaan Jaff
112295c58f bump: version 1.57.11 → 1.58.0 2025-01-12 22:02:40 -08:00
Ishaan Jaff
36c2883f6e
(proxy perf) - only read request body 1 time per request (#7728)
* req body

* fix linting
2025-01-12 22:00:59 -08:00
Ishaan Jaff
feee37e37f
fix svc logger (#7727) 2025-01-12 22:00:25 -08:00
Krrish Dholakia
e063c5a075 bump: version 1.57.10 → 1.57.11 2025-01-12 18:41:11 -08:00
Krish Dholakia
ec5a354eac
add azure o1 pricing (#7715)
* build(model_prices_and_context_window.json): add azure o1 pricing

Closes https://github.com/BerriAI/litellm/issues/7712

* refactor: replace regex with string method for whitespace check in stop-sequences handling (#7713)

* Allows overriding keep_alive time in ollama (#7079)

* Allows overriding keep_alive time in ollama

* Also adds to ollama_chat

* Adds some info on the docs about this parameter

* fix: together ai warning (#7688)

Co-authored-by: Carl Senze <carl.senze@aleph-alpha.com>

* fix(proxy_server.py): handle config containing thread locked objects when using get_config_state

* fix(proxy_server.py): add exception to debug

* build(model_prices_and_context_window.json): update 'supports_vision' for azure o1

---------

Co-authored-by: Wolfram Ravenwolf <52386626+WolframRavenwolf@users.noreply.github.com>
Co-authored-by: Regis David Souza Mesquita <github@rdsm.dev>
Co-authored-by: Carl <45709281+capsenz@users.noreply.github.com>
Co-authored-by: Carl Senze <carl.senze@aleph-alpha.com>
2025-01-12 18:15:35 -08:00
Ishaan Jaff
d4779deb0b
Revert "fix _read_request_body to re-use parsed body already (#7722)" (#7724)
This reverts commit 95183f2103.
2025-01-12 16:45:26 -08:00
Ishaan Jaff
b7e68eccdd fixes for img gen cost cal 2025-01-12 16:41:18 -08:00
Ishaan Jaff
c8cedbed20 fix img gen cost 2025-01-12 16:31:04 -08:00
Ishaan Jaff
b71021f1bf use set for public routes 2025-01-12 16:22:56 -08:00
Ishaan Jaff
bb1489eced fix optimize get llm provider 2025-01-12 16:21:23 -08:00
Ishaan Jaff
95183f2103
fix _read_request_body to re-use parsed body already (#7722) 2025-01-12 15:41:40 -08:00
Ishaan Jaff
2c25ea5737
(litellm sdk speedup) - use _model_contains_known_llm_provider in response_cost_calculator to check if the model contains a known litellm provider (#7721)
* define _cached_get_model_info_helper

* use _cached_get_model_info_helper

* speed up _select_model_name_for_cost_calc
2025-01-12 15:40:05 -08:00
Ishaan Jaff
6518bc70a0
(litellm SDK perf improvement) - use verbose_logger.debug and _cached_get_model_info_helper in _response_cost_calculator (#7720)
* define _cached_get_model_info_helper

* use _cached_get_model_info_helper
2025-01-12 15:27:54 -08:00
Ishaan Jaff
15b52039d2
(litellm sdk speedup router) - adds a helper _cached_get_model_group_info to use when trying to get deployment tpm/rpm limits (#7719)
* fix _cached_get_model_group_info

* fixes get_remaining_model_group_usage

* test_cached_get_model_group_info
2025-01-12 15:14:54 -08:00
Krrish Dholakia
baa528a102 bump: version 1.57.9 → 1.57.10 2025-01-11 23:30:53 -08:00
Krish Dholakia
ad2f66b3e3
[BETA] Add OpenAI /images/variations + Topaz API support (#7700)
* feat(main.py): initial commit for `/image/variations` endpoint support

* refactor(base_llm/): introduce new base llm base config for image variation endpoints

* refactor(openai/image_variations/transformation.py): implement openai image variation transformation handler

* fix: test

* feat(openai/): working openai `/image/variation` endpoint calls via sdk

* feat(topaz/): topaz sync image variation call support

Addresses https://github.com/BerriAI/litellm/issues/7593

'

* fix(topaz/transformation.py): fix linting errors

* fix(openai/image_variations/handler.py): fix passing json data

* fix(main.py): image_variation/

support async image variation route - `aimage_variation`

* fix(test_get_model_info.py): fix test

* fix: cleanup unused imports

* feat(openai/): add async `/image/variations` endpoint support

* feat(topaz/): support async `/image/variations` calls

* fix: test

* fix(utils.py): fix get_model_info_helper for no model info w/ provider config

handles situation where model info is not known but provider config exists

* test(test_router_fallbacks.py): mark flaky test

* fix: fix unused imports

* test: bump otel load test perf threshold - accounts for current load tests hitting same server
2025-01-11 23:27:46 -08:00
Ishaan Jaff
a7c803edc5
(perf) - only use response_cost_calculator 1 time per request. (Don't re-use the same helper twice per call ) (#7709)
* fix get_llm_provider for aiohttp openai

* fix _success_handler_helper_fn
2025-01-11 23:23:01 -08:00
Ishaan Jaff
75fb37217a
(sdk perf fix) - only print args passed to litellm when debugging mode is on (#7708)
* use _is_debugging_on

* fix unused imports
2025-01-11 22:56:20 -08:00
Ishaan Jaff
e50b389716 Merge branch 'litellm_aiohttp_openai_speedup' 2025-01-11 22:26:26 -08:00
Ishaan Jaff
7923cb1a64
fix _read_request_body (#7706) 2025-01-11 21:54:51 -08:00
Ishaan Jaff
1a6c4905f1 fix get_llm_provider for aiohttp openai 2025-01-11 21:45:06 -08:00
Ishaan Jaff
71e679246a
(perf sdk) - minor changes to cost calculator to run helpers only when necessary (#7704)
* use _get_model_info_helper

* fix _get_potential_model_names
2025-01-11 21:10:37 -08:00
Ishaan Jaff
dab7bebaf2
use _get_model_info_helper (#7703) 2025-01-11 21:08:15 -08:00
Krish Dholakia
becd4bc748
Litellm dev 01 11 2025 p3 (#7702)
* fix(__init__.py): fix init to exclude pricing-only model cost values from real model names

prevents bad health checks on wildcard routes

* fix(get_llm_provider.py): fix to handle calling bedrock_converse models
2025-01-11 20:06:54 -08:00
Krrish Dholakia
3062564488 docs(enterprise.md): cleanup docs and add faq 2025-01-11 10:46:55 -08:00
Krrish Dholakia
d988bfb6f8 docs(enterprise.md): clarify sla for patching vulnerabilities 2025-01-11 10:42:32 -08:00
Krish Dholakia
5e537fbdb1
fix(model_hub.tsx): clarify cost in model hub is per 1m tokens (#7687)
* fix(model_hub.tsx): clarify cost in model hub is per 1m tokens

* docs: test blog

* docs: improve release note docs

* docs(docs/): new stable release doc

* docs(docs/): specify date in all posts

* docs(docs/): add git diff to stable release docs
2025-01-11 09:57:09 -08:00
Krrish Dholakia
9a1c050cf7 docs: new release notes 2025-01-10 22:49:20 -08:00
Krrish Dholakia
f2ca244766 docs(logging.md): docs(logging.md): add docs on s3 bucket logging with team alias prefix 2025-01-10 22:28:05 -08:00
Krish Dholakia
599730960a
build: new ui build (#7685) 2025-01-10 22:12:17 -08:00
Krrish Dholakia
1f7f11f054 bump: version 1.57.8 → 1.57.9 2025-01-10 21:57:29 -08:00
Krish Dholakia
27892acdfc
Litellm dev 01 10 2025 p3 (#7682)
* feat(langfuse.py): log the used prompt when prompt management used

* test: fix test

* docs(self_serve.md): add doc on restricting personal key creation on ui

* feat(s3.py): support s3 logging with team alias prefixes (if available)

New preview feature

* fix(main.py): remove old if block - simplify to just await if coroutine returned

fixes lm_studio async embedding error

* fix(langfuse.py): handle get prompt check
2025-01-10 21:56:42 -08:00
Krish Dholakia
c4780479a9
Litellm dev 01 10 2025 p2 (#7679)
* test(test_basic_python_version.py): assert all optional dependencies are marked as extras on poetry

Fixes https://github.com/BerriAI/litellm/issues/7677

* docs(secret.md): clarify 'read_and_write' secret manager usage on aws

* docs(secret.md): fix doc

* build(ui/teams.tsx): add edit/delete button for updating user / team membership on ui

allows updating user role to admin on ui

* build(ui/teams.tsx): display edit member component on ui, when edit button on member clicked

* feat(team_endpoints.py): support updating team member role to admin via api endpoints

allows team member to become admin post-add

* build(ui/user_dashboard.tsx): if team admin - show all team keys

Fixes https://github.com/BerriAI/litellm/issues/7650

* test(config.yml): add tomli to ci/cd

* test: don't call python_basic_testing in local testing (covered by python 3.13 testing)
2025-01-10 21:50:53 -08:00
Ishaan Jaff
49d74748b0 fix showing release notes 2025-01-10 20:40:50 -08:00
Ishaan Jaff
189b677600 bump: version 1.57.7 → 1.57.8 2025-01-10 20:39:21 -08:00
Ishaan Jaff
02f5c44a35
[Bug fix]: Proxy Auth Layer - Allow Azure Realtime routes as llm_api_routes (#7684)
* fix route check azure realtime endpoints

* test_is_llm_api_route

* fix /realtime

* test_routes_on_litellm_proxy
2025-01-10 20:38:06 -08:00
Ishaan Jaff
2d1c90b688
fix proxy pre call hook - only use if user is using alerting (#7683) 2025-01-10 19:07:05 -08:00
Ishaan Jaff
9ac18caf24
uvicorn allow setting num workers (#7681) 2025-01-10 19:03:14 -08:00