ai-workspace-services/litellm
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
17,730 Commits 9 Branches 7 Tags 935 MiB
d3406c92aa
Commit Graph

17730 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Ishaan Jaff
d3406c92aa
[Feat] Log Request metadata on gcs bucket logging (#5743) 
* add requester_metadata in standard logging payload

* log requester_metadata in metadata

* use StandardLoggingPayload for logging

* docs StandardLoggingPayload

* fix import

* include standard logging object in failure

* add test for requester metadata
 2024-09-17 20:25:39 -07:00
Ishaan Jaff
1bb1f70a47
[Fix] Router/ Proxy - Tag Based routing, raise correct error when no deployments found and tag filtering is on (#5745) 
* fix tag routing - raise correct error when no model with tag based routing

* fix error string from tag based routing

* test router tag based routing

* raise 401 error when no tags avialable for deploymen

* linting fix
 2024-09-17 20:24:28 -07:00
Ishaan Jaff
911230c434
[Feat-Proxy-DataDog] Log Redis, Postgres Failure events on DataDog (#5750) 
* dd - start tracking redis status on dd

* add async_service_succes_hook / failure hook in custom logger

* add async_service_failure_hook

* log service failures on dd

* fix import error

* add test for redis errors / warning
 2024-09-17 20:24:06 -07:00
Ishaan Jaff
7f4dfe434a
[Fix] o1-mini causes pydantic warnings on reasoning_tokens (#5754) 
* add requester_metadata in standard logging payload

* log requester_metadata in metadata

* use StandardLoggingPayload for logging

* docs StandardLoggingPayload

* fix import

* include standard logging object in failure

* add test for requester metadata

* handle completion_tokens_details

* add test for completion_tokens_details
 2024-09-17 20:23:14 -07:00
dependabot[bot]
d0425e7767
Bump next from 14.1.1 to 14.2.10 in /ui/litellm-dashboard (#5753) 
Bumps [next](https://github.com/vercel/next.js) from 14.1.1 to 14.2.10.
- [Release notes](https://github.com/vercel/next.js/releases)
- [Changelog](https://github.com/vercel/next.js/blob/canary/release.js)
- [Commits](https://github.com/vercel/next.js/compare/v14.1.1...v14.2.10)

---
updated-dependencies:
- dependency-name: next
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2024-09-17 18:21:58 -07:00
Krish Dholakia
dd602753c0
Litellm fix router testing (#5748) 
* test: fix testing - azure changed content policy error logic

* test: fix tests to use mock responses

* test(test_image_generation.py): handle api instability

* test(test_image_generation.py): handle azure api instability

* fix(utils.py): fix unbounded variable error

* fix(utils.py): fix unbounded variable error

* test: refactor test to use mock response

* test: mark flaky azure tests
 2024-09-17 18:02:23 -07:00
Krrish Dholakia
8d4339c702 test(test_router_debug_logs.py): move to mock response 2024-09-17 11:38:47 -07:00
Ishaan Jaff
8de6e3d3ba
Revert "fix - deal with case when check view exists returns None (#5740)" (#5741) 
This reverts commit 535228159b.
 2024-09-17 09:04:22 -07:00
Ishaan Jaff
535228159b
fix - deal with case when check view exists returns None (#5740) 2024-09-17 08:38:19 -07:00
Krrish Dholakia
815d46f9e1 bump: version 1.46.1 → 1.46.2 2024-09-17 08:06:11 -07:00
Krish Dholakia
234185ec13
LiteLLM Minor Fixes & Improvements (09/16/2024) (#5723) (#5731) 
* LiteLLM Minor Fixes & Improvements (09/16/2024)  (#5723)

* coverage (#5713)

Signed-off-by: dbczumar <corey.zumar@databricks.com>

* Move (#5714)

Signed-off-by: dbczumar <corey.zumar@databricks.com>

* fix(litellm_logging.py): fix logging client re-init (#5710)

Fixes https://github.com/BerriAI/litellm/issues/5695

* fix(presidio.py): Fix logging_hook response and add support for additional presidio variables in guardrails config

Fixes https://github.com/BerriAI/litellm/issues/5682

* feat(o1_handler.py): fake streaming for openai o1 models

Fixes https://github.com/BerriAI/litellm/issues/5694

* docs: deprecated traceloop integration in favor of native otel (#5249)

* fix: fix linting errors

* fix: fix linting errors

* fix(main.py): fix o1 import

---------

Signed-off-by: dbczumar <corey.zumar@databricks.com>
Co-authored-by: Corey Zumar <39497902+dbczumar@users.noreply.github.com>
Co-authored-by: Nir Gazit <nirga@users.noreply.github.com>

* feat(spend_management_endpoints.py): expose `/global/spend/refresh` endpoint for updating material view (#5730)

* feat(spend_management_endpoints.py): expose `/global/spend/refresh` endpoint for updating material view

Supports having `MonthlyGlobalSpend` view be a material view, and exposes an endpoint to refresh it

* fix(custom_logger.py): reset calltype

* fix: fix linting errors

* fix: fix linting error

* fix: fix import

* test(test_databricks.py): fix databricks tests

---------

Signed-off-by: dbczumar <corey.zumar@databricks.com>
Co-authored-by: Corey Zumar <39497902+dbczumar@users.noreply.github.com>
Co-authored-by: Nir Gazit <nirga@users.noreply.github.com>
 2024-09-17 08:05:52 -07:00
Ishaan Jaff
1e59395280 fix guardrail linting change 2024-09-16 20:12:54 -07:00
Ishaan Jaff
7e9dbcd1a3 fix gemini 1.5 flash test 2024-09-16 19:37:41 -07:00
Ishaan Jaff
8762b64b1d ci/cd run again 2024-09-16 18:26:53 -07:00
Ishaan Jaff
9f5a33015f fix linting 2024-09-16 18:07:48 -07:00
Ishaan Jaff
3b034224a0 bump: version 1.46.0 → 1.46.1 2024-09-16 18:06:34 -07:00
Ishaan Jaff
b6ae2204a8
[Feat-Proxy] Slack Alerting - allow using os.environ/ vars for alert to webhook url (#5726) 
* allow using os.environ for slack urls

* use env vars for webhook urls

* fix types for get_secret

* fix linting

* fix linting

* fix linting

* linting fixes

* linting fix

* docs alerting slack

* fix get data
 2024-09-16 18:03:37 -07:00
Ishaan Jaff
8103e2b2da
[Fix-Proxy] Azure Key Management - Secret Manager (#5728) 
* fix azure key mgtm error

* add test for azure kms

* add test for azure kms
 2024-09-16 18:01:40 -07:00
Ishaan Jaff
ca6d99e1ad fix gemini 1.5 flash supports_response_schema 2024-09-16 17:59:59 -07:00
Ishaan Jaff
ce7a937a53 fix test_all_model_config 2024-09-16 17:47:31 -07:00
Ishaan Jaff
4dcb092d12 fix test_all_model_configs 2024-09-16 17:44:48 -07:00
Ishaan Jaff
7b09591ca6
[Fix-Proxy] log exceptions from azure key vault on verbose_logger.exceptions (#5719) 
* log exceptions from azure key vault

* fix error from azure key vault
 2024-09-16 16:58:37 -07:00
Ishaan Jaff
8fbe2abb89
[Feat-Proxy] Add upperbound key duration param (#5727) 
* add upperbound key duration param

* use upper bound values when None set

* docs upperbound params
 2024-09-16 16:28:36 -07:00
Ishaan Jaff
3a5039e284
Warning fix for Pydantic 2.0 (#5679) (#5707) 
* Fixed UserWarning: Valid config keys have changed in V2 underscore_attrs_are_private

* Trying different method

Co-authored-by: CyanideByte <cyanidebyte@hotmail.com>
 2024-09-16 11:24:12 -07:00
David Manouchehri
aa64c34ce6
Add unsupported params. (#5722) 2024-09-16 09:43:50 -07:00
Krrish Dholakia
3c741b7beb docs(docker_quick_start.md): update quick start with azure connection error 2024-09-16 07:31:32 -07:00
Krrish Dholakia
5fb270a559 build(model_prices_and_context_window.json): bump claude-3-5-sonnet max tokens 2024-09-15 13:57:41 -07:00
F1bos
b64b7a94ae
(models): Enable JSON Schema Support for Gemini 1.5 Flash Models (#5708) 
* Fixed gemini-1.5-flash pricing

* (models): Added missing gemini experimental models + fixed pricing for gemini-1.5-pro-exp-0827

* Added gemini/gemini-1.5-flash-001 model

* Updated supports_response_schema to true for gemini flash 1.5 models
 2024-09-15 13:52:00 -07:00
Krish Dholakia
da77706c26
Litellm stable dev (#5711) 
* feat(aws_base_llm.py): prevents recreating boto3 credentials during high traffic

Leads to 100ms perf boost in local testing

* fix(base_aws_llm.py): fix credential caching check to see if token is set

* refactor(bedrock/chat): separate converse api and invoke api + isolate converse api transformation logic

Make it easier to see how requests are transformed for /converse

* fix: fix imports

* fix(bedrock/embed): fix reordering of headers

* fix(base_aws_llm.py): fix get credential logic

* fix(converse_handler.py): fix ai21 streaming response
 2024-09-14 23:22:59 -07:00
Ishaan Jaff
2efdd2a6a4 mark test as flaky 2024-09-14 19:32:22 -07:00
Ishaan Jaff
0c33b8dd12 docs 2024-09-14 19:13:45 -07:00
Ishaan Jaff
c220fc0e92 docs max_completion_tokens 2024-09-14 19:12:12 -07:00
Ishaan Jaff
e447784650 bump: version 1.45.0 → 1.46.0 2024-09-14 18:49:24 -07:00
Ishaan Jaff
680d00ed11
[Feat-Prometheus] Add prometheus metric for tracking cooldown events (#5705) 
* add litellm_deployment_cooled_down

* track num cooldowns on prometheus

* track exception status

* fix linting

* docs prom metrics

* cleanup premium user checks
 2024-09-14 18:46:45 -07:00
Ishaan Jaff
c8eff2dc65
[Feat-Prometheus] Track exception status on litellm_deployment_failure_responses (#5706) 
* add litellm_deployment_cooled_down

* track num cooldowns on prometheus

* track exception status

* fix linting

* docs prom metrics

* cleanup premium user checks

* prom track deployment failure state

* docs prometheus
 2024-09-14 18:44:31 -07:00
Ishaan Jaff
b878a67a7c fic otel load test % 2024-09-14 18:04:28 -07:00
Ishaan Jaff
c8d15544c8
[Fix] Router cooldown logic - use % thresholds instead of allowed fails to cooldown deployments (#5698) 
* move cooldown logic to it's own helper

* add new track deployment metrics folder

* increment success, fails for deployment in current minute

* fix cooldown logic

* fix test_aaarouter_dynamic_cooldown_message_retry_time

* fix test_single_deployment_no_cooldowns_test_prod_mock_completion_calls

* clean up get from deployment test

* fix _async_get_healthy_deployments

* add mock InternalServerError

* test deployment failing 25% requests

* add test_high_traffic_cooldowns_one_bad_deployment

* fix vertex load test

* add test for rate limit error models in cool down

* change default cooldown time

* fix cooldown message time

* fix cooldown on 429 error

* fix doc string for _should_cooldown_deployment

* fix sync cooldown logic router
 2024-09-14 18:01:19 -07:00
Ishaan Jaff
7c2ddba6c6
sambanova support (#5547) (#5703) 
* add sambanova support

* sambanova support

* updated api endpoint for sambanova

---------

Co-authored-by: Venu Anuganti <venu@venublog.com>
Co-authored-by: Venu Anuganti <venu@vairmac2020>
 2024-09-14 17:23:04 -07:00
Ishaan Jaff
85acdb9193
[Feat] Add max_completion_tokens param (#5691) 
* add max_completion_tokens

* add max_completion_tokens

* add max_completion_tokens support for OpenAI models

* add max_completion_tokens param

* add max_completion_tokens for bedrock converse models

* add test for converse maxTokens

* fix openai o1 param mapping test

* move test optional params

* add max_completion_tokens for anthropic api

* fix conftest

* add max_completion tokens for vertex ai partner models

* add max_completion_tokens for fireworks ai

* add max_completion_tokens for hf rest api

* add test for param mapping

* add param mapping for vertex, gemini + testing

* predibase is the most unstable and unusable llm api in prod, can't handle our ci/cd

* add max_completion_tokens to openai supported params

* fix fireworks ai param mapping
 2024-09-14 14:57:01 -07:00
Ahmet
415a3ede9e
Update model_prices_and_context_window.json (#5700) 
added audio_speech mode on the sample_spec for clarity.
 2024-09-14 11:22:08 -07:00
Krish Dholakia
dad1ad2077
LiteLLM Minor Fixes and Improvements (09/14/2024) (#5697) 
* fix(health_check.py): hide sensitive keys from health check debug information k

* fix(route_llm_request.py): fix proxy model not found error message to indicate how to resolve issue

* fix(vertex_llm_base.py): fix exception message to not log credentials
 2024-09-14 10:32:39 -07:00
Krish Dholakia
60709a0753
LiteLLM Minor Fixes and Improvements (09/13/2024) (#5689) 
* refactor: cleanup unused variables + fix pyright errors

* feat(health_check.py): Closes https://github.com/BerriAI/litellm/issues/5686

* fix(o1_reasoning.py): add stricter check for o-1 reasoning model

* refactor(mistral/): make it easier to see mistral transformation logic

* fix(openai.py): fix openai o-1 model param mapping

Fixes https://github.com/BerriAI/litellm/issues/5685

* feat(main.py): infer finetuned gemini model from base model

Fixes https://github.com/BerriAI/litellm/issues/5678

* docs(vertex.md): update docs to call finetuned gemini models

* feat(proxy_server.py): allow admin to hide proxy model aliases

Closes https://github.com/BerriAI/litellm/issues/5692

* docs(load_balancing.md): add docs on hiding alias models from proxy config

* fix(base.py): don't raise notimplemented error

* fix(user_api_key_auth.py): fix model max budget check

* fix(router.py): fix elif

* fix(user_api_key_auth.py): don't set team_id to empty str

* fix(team_endpoints.py): fix response type

* test(test_completion.py): handle predibase error

* test(test_proxy_server.py): fix test

* fix(o1_transformation.py): fix max_completion_token mapping

* test(test_image_generation.py): mark flaky test
 2024-09-14 10:02:55 -07:00
F1bos
db3af20d84
(models): Added missing gemini experimental models + fixed pricing for gemini-1.5-pro-exp-0827 (#5693) 
* Fixed gemini-1.5-flash pricing

* (models): Added missing gemini experimental models + fixed pricing for gemini-1.5-pro-exp-0827
 2024-09-14 08:41:48 -07:00
Ishaan Jaff
741c8e8a45
[Feat - Perf Improvement] DataDog Logger 91% lower latency (#5687) 
* fix refactor dd to be an instance of custom logger

* migrate dd logger to be async

* clean up dd logging

* add datadog sync and async code

* use batching for datadog logger

* add doc string for dd logging

* add clear doc string

* fix doc string

* allow debugging intake url

* clean up requirements.txt

* allow setting custom batch size on logger

* fix dd logging to use compression

* fix linting

* add dd load test

* fix dd load test

* fix dd url

* add test_datadog_logging_http_request

* fix test_datadog_logging_http_request
 2024-09-13 17:39:17 -07:00
Ishaan Jaff
cd8d7ca915
[Fix] Performance - use in memory cache when downloading images from a url (#5657) 
* fix use in memory cache when getting images

* fix linting

* fix load testing

* fix load test size

* fix load test size

* trigger ci/cd again
 2024-09-13 07:23:42 -07:00
Krrish Dholakia
cdd7cd4d69 build: bump from 1.44.28 -> 1.45.0 2024-09-12 23:10:29 -07:00
Krish Dholakia
4657a40ef1
LiteLLM Minor Fixes and Improvements (09/12/2024) (#5658) 
* fix(factory.py): handle tool call content as list

Fixes https://github.com/BerriAI/litellm/issues/5652

* fix(factory.py): enforce stronger typing

* fix(router.py): return model alias in /v1/model/info and /v1/model_group/info

* fix(user_api_key_auth.py): move noisy warning message to debug

cleanup logs

* fix(types.py): cleanup pydantic v2 deprecated param

Fixes https://github.com/BerriAI/litellm/issues/5649

* docs(gemini.md): show how to pass inline data to gemini api

Fixes https://github.com/BerriAI/litellm/issues/5674
 2024-09-12 23:04:06 -07:00
David Manouchehri
795047c37f
Add o1 models on OpenRouter. (#5676) 2024-09-12 22:16:10 -07:00
Krish Dholakia
00047de1c6
fix(user_dashboard.tsx): don't call /global/spend on startup (#5668) 
at 1m+ rows, query timeouts cause ui errors
 2024-09-12 22:15:52 -07:00
Krish Dholakia
d94d47424f
fix(proxy/utils.py): auto-update if required view missing from db. raise warning for optional views. (#5675) 
Prevents missing optional views from blocking proxy startup.
 2024-09-12 22:15:44 -07:00