ai-workspace-services/litellm
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
05aa740a7d
litellm/tests
History
Krish Dholakia 89fcd7b0e1
Litellm 12 02 2024 (#6994) 
* add the logprobs param for fireworks ai (#6915)

* add the logprobs param for fireworks ai

* (feat) pass through llm endpoints - add `PATCH` support (vertex context caching requires for update ops)  (#6924)

* add PATCH for pass through endpoints

* test_pass_through_routes_support_all_methods

* sonnet supports pdf, haiku does not (#6928)

* (feat) DataDog Logger - Add Failure logging + use Standard Logging payload (#6929)

* add async_log_failure_event for dd

* use standard logging payload for DD logging

* use standard logging payload for DD

* fix use SLP status

* allow opting into _create_v0_logging_payload

* add unit tests for DD logging payload

* fix dd logging tests

* (feat) log proxy auth errors on datadog  (#6931)

* add new dd type for auth errors

* add async_log_proxy_authentication_errors

* fix comment

* use async_log_proxy_authentication_errors

* test_datadog_post_call_failure_hook

* test_async_log_proxy_authentication_errors

* (feat) Allow using include to include external YAML files in a config.yaml (#6922)

* add helper to process inlcudes directive on yaml

* add doc on config management

* unit tests for `include` on config.yaml

* bump: version 1.52.16 → 1.53.

* (feat) dd logger - set tags according to the values set by those env vars  (#6933)

* dd logger, inherit from .envs

* test_datadog_payload_environment_variables

* fix _get_datadog_service

* build(ui/): update ui build

* bump: version 1.53.0 → 1.53.1

* Revert "(feat) Allow using include to include external YAML files in a config.yaml (#6922)"

This reverts commit 68e59824a3.

* LiteLLM Minor Fixes & Improvements (11/26/2024)  (#6913)

* docs(config_settings.md): document all router_settings

* ci(config.yml): add router_settings doc test to ci/cd

* test: debug test on ci/cd

* test: debug ci/cd test

* test: fix test

* fix(team_endpoints.py): skip invalid team object. don't fail `/team/list` call

Causes downstream errors if ui just fails to load team list

* test(base_llm_unit_tests.py): add 'response_format={"type": "text"}' test to base_llm_unit_tests

adds complete coverage for all 'response_format' values to ci/cd

* feat(router.py): support wildcard routes in `get_router_model_info()`

Addresses https://github.com/BerriAI/litellm/issues/6914

* build(model_prices_and_context_window.json): add tpm/rpm limits for all gemini models

Allows for ratelimit tracking for gemini models even with wildcard routing enabled

Addresses https://github.com/BerriAI/litellm/issues/6914

* feat(router.py): add tpm/rpm tracking on success/failure to global_router

Addresses https://github.com/BerriAI/litellm/issues/6914

* feat(router.py): support wildcard routes on router.get_model_group_usage()

* fix(router.py): fix linting error

* fix(router.py): implement get_remaining_tokens_and_requests

Addresses https://github.com/BerriAI/litellm/issues/6914

* fix(router.py): fix linting errors

* test: fix test

* test: fix tests

* docs(config_settings.md): add missing dd env vars to docs

* fix(router.py): check if hidden params is dict

* LiteLLM Minor Fixes & Improvements (11/27/2024) (#6943)

* fix(http_parsing_utils.py): remove `ast.literal_eval()` from http utils

Security fix - https://huntr.com/bounties/96a32812-213c-4819-ba4e-36143d35e95b?token=bf414bbd77f8b346556e
64ab2dd9301ea44339910877ea50401c76f977e36cdd78272f5fb4ca852a88a7e832828aae1192df98680544ee24aa98f3cf6980d8
bab641a66b7ccbc02c0e7d4ddba2db4dbe7318889dc0098d8db2d639f345f574159814627bb084563bad472e2f990f825bff0878a9
e281e72c88b4bc5884d637d186c0d67c9987c57c3f0caf395aff07b89ad2b7220d1dd7d1b427fd2260b5f01090efce5250f8b56ea2
c0ec19916c24b23825d85ce119911275944c840a1340d69e23ca6a462da610

* fix(converse/transformation.py): support bedrock apac cross region inference

Fixes https://github.com/BerriAI/litellm/issues/6905

* fix(user_api_key_auth.py): add auth check for websocket endpoint

Fixes https://github.com/BerriAI/litellm/issues/6926

* fix(user_api_key_auth.py): use `model` from query param

* fix: fix linting error

* test: run flaky tests first

* docs: update the docs (#6923)

* (bug fix) /key/update was not storing `budget_duration` in the DB  (#6941)

* fix - store budget_duration for keys

* test_generate_and_update_key

* test_update_user_unit_test

* fix user update

* (fix) handle json decode errors for DD exception logging (#6934)

* fix JSONDecodeError

* handle async_log_proxy_authentication_errors

* fix test_async_log_proxy_authentication_errors_get_request

* Revert "Revert "(feat) Allow using include to include external YAML files in a config.yaml (#6922)""

This reverts commit 5d13302e6b.

* (docs + fix) Add docs on Moderations endpoint, Text Completion  (#6947)

* fix _pass_through_moderation_endpoint_factory

* fix route_llm_request

* doc moderations api

* docs on /moderations

* add e2e tests for moderations api

* docs moderations api

* test_pass_through_moderation_endpoint_factory

* docs text completion

* (feat) add enforcement for unique key aliases on /key/update and /key/generate  (#6944)

* add enforcement for unique key aliases

* fix _enforce_unique_key_alias

* fix _enforce_unique_key_alias

* fix _enforce_unique_key_alias

* test_enforce_unique_key_alias

* (fix) tag merging / aggregation logic   (#6932)

* use 1 helper to merge tags + ensure unique ness

* test_add_litellm_data_to_request_duplicate_tags

* fix _merge_tags

* fix proxy utils test

* fix doc string

* (feat) Allow disabling ErrorLogs written to the DB  (#6940)

* fix - allow disabling logging error logs

* docs on disabling error logs

* doc string for _PROXY_failure_handler

* test_disable_error_logs

* rename file

* fix rename file

* increase test coverage for test_enable_error_logs

* fix(key_management_endpoints.py): support 'tags' param on `/key/update` (#6945)

* LiteLLM Minor Fixes & Improvements (11/29/2024)  (#6965)

* fix(factory.py): ensure tool call converts image url

Fixes https://github.com/BerriAI/litellm/issues/6953

* fix(transformation.py): support mp4 + pdf url's for vertex ai

Fixes https://github.com/BerriAI/litellm/issues/6936

* fix(http_handler.py): mask gemini api key in error logs

Fixes https://github.com/BerriAI/litellm/issues/6963

* docs(prometheus.md): update prometheus FAQs

* feat(auth_checks.py): ensure specific model access > wildcard model access

if wildcard model is in access group, but specific model is not - deny access

* fix(auth_checks.py): handle auth checks for team based model access groups

handles scenario where model access group used for wildcard models

* fix(internal_user_endpoints.py): support adding guardrails on `/user/update`

Fixes https://github.com/BerriAI/litellm/issues/6942

* fix(key_management_endpoints.py): fix prepare_metadata_fields helper

* fix: fix tests

* build(requirements.txt): bump openai dep version

fixes proxies argument

* test: fix tests

* fix(http_handler.py): fix error message masking

* fix(bedrock_guardrails.py): pass in prepped data

* test: fix test

* test: fix nvidia nim test

* fix(http_handler.py): return original response headers

* fix: revert maskedhttpstatuserror

* test: update tests

* test: cleanup test

* fix(key_management_endpoints.py): fix metadata field update logic

* fix(key_management_endpoints.py): maintain initial order of guardrails in key update

* fix(key_management_endpoints.py): handle prepare metadata

* fix: fix linting errors

* fix: fix linting errors

* fix: fix linting errors

* fix: fix key management errors

* fix(key_management_endpoints.py): update metadata

* test: update test

* refactor: add more debug statements

* test: skip flaky test

* test: fix test

* fix: fix test

* fix: fix update metadata logic

* fix: fix test

* ci(config.yml): change db url for e2e ui testing

* bump: version 1.53.1 → 1.53.2

* Updated config.yml

---------

Co-authored-by: Ishaan Jaff <ishaanjaffer0324@gmail.com>
Co-authored-by: paul-gauthier <69695708+paul-gauthier@users.noreply.github.com>
Co-authored-by: Krrish Dholakia <krrishdholakia@gmail.com>
Co-authored-by: Sara Han <127759186+sdiazlor@users.noreply.github.com>

* fix(exceptions.py): ensure ratelimit error code == 429, type == "throttling_error"

Fixes https://github.com/BerriAI/litellm/pull/6973

* fix(utils.py): add jina ai dimensions embedding param support

Fixes https://github.com/BerriAI/litellm/issues/6591

* fix(exception_mapping_utils.py): add bedrock 'prompt is too long' exception to context window exceeded error exception mapping

Fixes https://github.com/BerriAI/litellm/issues/6629

Closes https://github.com/BerriAI/litellm/pull/6975

* fix(litellm_logging.py): strip trailing slash for api base

Closes https://github.com/BerriAI/litellm/pull/6859

* test: skip timeout issue

---------

Co-authored-by: ershang-dou <erlie.shang@gmail.com>
Co-authored-by: Ishaan Jaff <ishaanjaffer0324@gmail.com>
Co-authored-by: paul-gauthier <69695708+paul-gauthier@users.noreply.github.com>
Co-authored-by: Sara Han <127759186+sdiazlor@users.noreply.github.com>
2024-12-02 22:00:01 -08:00
..
basic_proxy_startup_tests (fix) don't block proxy startup if license check fails & using prometheus (#6839) 2024-11-20 17:55:39 -08:00
code_coverage_tests (Perf / latency improvement) improve pass through endpoint latency to ~50ms (before PR was 400ms) (#6874) 2024-11-22 18:47:26 -08:00
documentation_tests LiteLLM Minor Fixes & Improvements (11/26/2024) (#6913) 2024-11-28 00:01:38 +05:30
image_gen_tests (fix) add linting check to ban creating AsyncHTTPHandler during LLM calling (#6855) 2024-11-21 19:03:02 -08:00
llm_translation Litellm 12 02 2024 (#6994) 2024-12-02 22:00:01 -08:00
load_tests (load testing) add vertex_ai embeddings load test (#6004) 2024-10-03 14:39:15 +05:30
local_testing Litellm 12 02 2024 (#6994) 2024-12-02 22:00:01 -08:00
logging_callback_tests Litellm 12 02 2024 (#6994) 2024-12-02 22:00:01 -08:00
old_proxy_tests/tests Litellm ruff linting enforcement (#5992) 2024-10-01 19:44:20 -04:00
otel_tests LiteLLM Minor Fixes & Improvements (11/29/2024) (#6965) 2024-12-01 05:24:11 -08:00
pass_through_tests (fix) pass through endpoints - run logging async + use thread pool executor for sync logging callbacks (#6907) 2024-11-25 22:52:05 -08:00
pass_through_unit_tests (feat) pass through llm endpoints - add PATCH support (vertex context caching requires for update ops) (#6924) 2024-11-26 14:39:13 -08:00
proxy_admin_ui_tests LiteLLM Minor Fixes & Improvements (11/29/2024) (#6965) 2024-12-01 05:24:11 -08:00
proxy_unit_tests LiteLLM Minor Fixes & Improvements (11/29/2024) (#6965) 2024-12-01 05:24:11 -08:00
router_unit_tests (docs + fix) Add docs on Moderations endpoint, Text Completion (#6947) 2024-11-27 16:30:48 -08:00
gettysburg.wav
large_text.py
openai_batch_completions.jsonl feat(router.py): Support Loadbalancing batch azure api endpoints (#5469) 2024-09-02 21:32:55 -07:00
README.MD
test_callbacks_on_proxy.py
test_config.py test_team_logging 2024-11-21 22:01:12 -08:00
test_debug_warning.py fix(utils.py): fix togetherai streaming cost calculation 2024-08-01 15:03:08 -07:00
test_end_users.py Litellm ruff linting enforcement (#5992) 2024-10-01 19:44:20 -04:00
test_entrypoint.py (fix) clean up root repo - move entrypoint.sh and build_admin_ui to /docker (#6110) 2024-10-08 11:34:43 +05:30
test_fallbacks.py fix(user_api_key_auth.py): ensure user has access to fallback models 2024-06-20 16:02:19 -07:00
test_health.py
test_keys.py LiteLLM Minor Fixes & Improvements (11/29/2024) (#6965) 2024-12-01 05:24:11 -08:00
test_logging.conf feat(proxy_cli.py): add new 'log_config' cli param (#6352) 2024-10-21 21:25:58 -07:00
test_models.py Litellm ruff linting enforcement (#5992) 2024-10-01 19:44:20 -04:00
test_openai_batches_endpoint.py test batches endpoint on proxy 2024-07-30 09:46:30 -07:00
test_openai_endpoints.py fix(lowest_tpm_rpm_routing.py): fix parallel rate limit check (#6577) 2024-11-05 22:03:44 +05:30
test_openai_files_endpoints.py test - batches endpoint 2024-07-26 18:09:49 -07:00
test_openai_fine_tuning.py fix cancel ft job route 2024-07-31 16:19:15 -07:00
test_organizations.py test: skip flaky test 2024-11-22 19:23:36 +05:30
test_passthrough_endpoints.py test test_basic_passthrough 2024-08-06 21:17:07 -07:00
test_ratelimit.py
test_spend_logs.py LiteLLM Minor Fixes & Improvements (11/29/2024) (#6965) 2024-12-01 05:24:11 -08:00
test_team_logging.py test: skip flaky test 2024-11-22 19:23:36 +05:30
test_team.py Litellm dev 11 08 2024 (#6658) 2024-11-08 22:07:17 +05:30
test_users.py LiteLLM Minor Fixes & Improvements (10/08/2024) (#6119) 2024-10-08 21:57:03 -07:00

README.MD

In total litellm runs 500+ tests Most tests are in /litellm/tests. These are just the tests for the proxy docker image, used for circle ci.