ai-workspace-services/litellm
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
fb7091ef79
litellm/tests/local_testing
History
Mateo Wang 2b00ea9ee4
test(ci): skip Fireworks tests on 404 + Gemini image-size test on 429 
Four pre-existing flakes on main that gate this branch's workflow even
though they're unrelated to the reasoning_effort_grid suite:

1. tests/local_testing/test_completion.py::test_completion_fireworks_ai
2. tests/local_testing/test_completion_cost.py::test_completion_cost_fireworks_ai[fireworks_ai/llama-v3p3-70b-instruct]
3. tests/llm_translation/test_fireworks_ai_translation.py::test_document_inlining_example[False]

   The Fireworks-hosted `llama-v3p3-70b-instruct` deployment is currently
   returning 404 "Model not found, inaccessible, and/or not deployed".
   These tests pass when the model is deployed; the issue is upstream
   capacity, not our code path. Wrap the live call in a try/except that
   pytest.skip's on litellm.NotFoundError so a Fireworks deployment hiccup
   no longer fails CI for unrelated PRs.

4. tests/llm_translation/test_gemini.py::test_gemini_image_size_limit_exceeded

   The test fetches the 32MB "Blue Marble 2002" image from Wikimedia to
   exercise the 50MB image-size cap. CI runners share an IP pool with
   noisy traffic, so Wikimedia routinely returns HTTP 429. The size-limit
   check never gets a chance to fire. Catch the 429 BadRequestError and
   pytest.skip in that case.

None of these belong on this PR conceptually, but they're included per
request to unblock the workflow before morning.
2026-05-16 07:47:25 +00:00
..
.litellm_cache
auto_router
example_config_yaml test: test 2026-03-28 19:17:38 -07:00
test_configs test: test 2026-03-28 19:17:38 -07:00
test_model_response_typing
azure_fine_tune.jsonl
azure_speech.mp3 [Feat] Add Azure AVA TTS integration (#15749) 2025-10-20 16:52:23 -07:00
batch_job_results_furniture.jsonl
cache_unit_tests.py fix: use fastuuid helper (#14903) 2025-09-25 15:47:01 -07:00
conftest.py test(vcr): classify cache verdicts, detect live calls, surface cost leaks 2026-05-13 00:31:47 +00:00
create_mock_standard_logging_payload.py chore(ci): modernize model references in tests and configs (#27856) 2026-05-15 15:44:28 -07:00
data_map.txt
eagle.wav
example.jsonl
gettysburg.wav
large_text.py
model_cost.json
openai_batch_completions_router.jsonl
openai_batch_completions.jsonl
speech_vertex.mp3
stream_chunk_testdata.py
test_acompletion_fallbacks.py
test_acompletion.py
test_acooldowns_router.py test: test 2026-03-28 19:17:38 -07:00
test_add_function_to_prompt.py
test_add_update_models.py fix(tests): skip remaining real prisma DB tests in CI and related test suites 2026-02-20 13:25:42 -03:00
test_aim_guardrails.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_alangfuse.py test: update key names 2026-03-28 21:13:16 -07:00
test_amazing_vertex_completion.py fix(vertex): synthesize items for array types missing items entirely 2026-04-28 09:23:55 -07:00
test_anthropic_prompt_caching.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_arize_ai.py test: rename env var 2026-03-28 20:27:39 -07:00
test_arize_phoenix.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_assistants.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_async_fn.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_auth_utils.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_azure_anthropic_sync_post.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_azure_content_safety.py
test_azure_openai.py test: test 2026-03-28 19:17:38 -07:00
test_azure_perf.py test: test 2026-03-28 19:17:38 -07:00
test_basic_python_version.py [Test] CI: add v2 migration resolver coverage with local Postgres 2026-04-21 14:40:11 -07:00
test_batch_completion_return_exceptions.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_batch_completions.py replace retired claude-3-haiku-20240307 with claude-haiku-4-5-20251001 in local_testing part1 and router fallback tests 2026-04-20 16:10:45 -07:00
test_blocked_user_list.py fix(tests): skip remaining real prisma DB tests in CI and related test suites 2026-02-20 13:25:42 -03:00
test_braintrust.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_budget_manager.py
test_cache_preset_key.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_caching_handler.py fix(caching): defer streaming cache-hit callbacks for all stream=True 2026-05-01 17:03:32 +05:30
test_caching_ssl.py Merge main and resolve conflict in test_router_client_init.py 2026-03-30 18:44:33 -07:00
test_caching.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_class.py test: test 2026-03-28 19:17:38 -07:00
test_completion_cost.py test(ci): skip Fireworks tests on 404 + Gemini image-size test on 429 2026-05-16 07:47:25 +00:00
test_completion_with_retries.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_completion.py test(ci): skip Fireworks tests on 404 + Gemini image-size test on 429 2026-05-16 07:47:25 +00:00
test_config.py test: test 2026-03-28 19:17:38 -07:00
test_cost_calc.py Revert "Fix xdist test isolation: capture true defaults and poll instead of sleep" 2026-03-15 22:57:39 -07:00
test_custom_api_logger.py
test_custom_callback_input.py fix(tests): drop dall-e-only test classes; route live image tests via gpt-image-1 2026-05-12 16:16:59 -07:00
test_custom_llm.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_custom_logger.py Mark test_redis_cache_completion_stream as flaky with retries 2026-03-15 20:44:18 -07:00
test_disk_cache_unit_tests.py
test_docker_no_network_on_deploy.py build: migrate packaging, CI, and Docker from Poetry to uv (#25007) 2026-04-09 11:46:23 -07:00
test_dual_cache.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_dynamic_rate_limit_handler.py fix: use fastuuid helper (#14903) 2025-09-25 15:47:01 -07:00
test_dynamodb_logs.py
test_embedding.py [Test] Tests: Stop parametrizing API keys into pytest test IDs (#27249) 2026-05-05 17:21:18 -07:00
test_exceptions.py fix: cleanup tests 2026-03-30 16:24:35 -07:00
test_file_types.py
test_function_call_parsing.py replace retired claude-3-haiku-20240307 with claude-haiku-4-5-20251001 in local_testing part1 and router fallback tests 2026-04-20 16:10:45 -07:00
test_function_calling.py fix(anthropic): inject dummy tool without modify_params (#27620) 2026-05-11 09:50:16 -07:00
test_function_setup.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_gcs_bucket.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_gcs_cache_unit_tests.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_gemini_reasoning_content.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_get_llm_provider.py test: drop duplicate openrouter prefix-strip test 2026-04-25 18:06:25 -03:00
test_get_model_file.py Revert "Merge pull request #16590 from Chesars/refactor/remove-backup-file-dry-principle" 2026-04-25 17:10:41 -03:00
test_get_model_info.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_get_optional_params_embeddings.py fix(embeddings): allow dimensions param passthrough via allowed_openai_params for non-text-embedding-3 OpenAI models 2026-02-26 09:59:37 +05:30
test_get_optional_params_functions_not_supported.py
test_google_ai_studio_gemini.py
test_guardrails_ai.py
test_helicone_integration.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_http_parsing_utils.py
test_img_resize.py
test_lakera_ai_prompt_injection.py
test_langchain_ChatLiteLLM.py
test_langsmith.py fix: use fastuuid helper (#14903) 2025-09-25 15:47:01 -07:00
test_least_busy_routing.py test: fixes because azure deactivated our account 2025-10-25 15:10:45 -07:00
test_litellm_max_budget.py
test_llm_guard.py
test_load_test_router_s3.py fix tests 2025-10-25 10:19:24 -07:00
test_loadtest_router.py test: test 2026-03-28 19:17:38 -07:00
test_logfire.py
test_logging.py
test_longer_context_fallback.py
test_lowest_cost_routing.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_lowest_latency_routing.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_lunary.py
test_max_tpm_rpm_limiter.py
test_mem_leak.py
test_mem_usage.py fix tests 2025-10-25 10:19:24 -07:00
test_mock_request.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_model_alias_map.py fix(test): scope ERROR log assertion to LiteLLM logger in test_model_alias_map 2026-04-29 03:48:41 +00:00
test_model_max_token_adjust.py
test_multiple_deployments.py [Fix] TogetherAIConfig.get_supported_openai_params recursion 2026-04-16 17:20:58 -07:00
test_ollama_local_chat.py
test_ollama_local.py
test_ollama.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_openai_moderations_hook.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_opik.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_pass_through_endpoints.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_profiling_router.py
test_prometheus_service.py [Release Fix] (#22411) 2026-02-28 09:46:35 -08:00
test_prompt_caching.py claude-sonnet-4-5-20250929 fix 2025-10-31 18:20:52 -07:00
test_prompt_injection_detection.py test: test 2026-03-28 19:17:38 -07:00
test_promptlayer_integration.py
test_provider_specific_config.py Litellm fix update bedrock models (#24947) 2026-04-01 19:22:54 -07:00
test_pydantic_namespaces.py
test_pydantic.py
test_redis_batch_optimizations.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_register_model.py Revert "test_update_model_cost_map_url" 2025-12-22 12:41:30 +05:30
test_responses_stream_cache_keys.py fix(cache): persist and replay streamed Responses API requests (#24580) 2026-05-01 11:55:36 +05:30
test_router_auto_router.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_router_batch_completion.py
test_router_budget_limiter.py test: test 2026-03-28 19:17:38 -07:00
test_router_caching.py test: test 2026-03-28 19:17:38 -07:00
test_router_client_init.py test_router_init_azure_service_principal_with_secret_with_environment_variables 2026-03-30 21:15:53 -07:00
test_router_cooldown_handlers.py test: test 2026-03-28 19:17:38 -07:00
test_router_custom_routing.py Optimize CI: parallelize router and guardrails test jobs, fix test isolation 2026-03-14 22:54:44 -07:00
test_router_debug_logs.py feat: routing groups ui 2026-05-04 18:09:14 -07:00
test_router_fallback_handlers.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_router_fallbacks.py replace retired claude-3-haiku-20240307 with claude-haiku-4-5-20251001 in local_testing part1 and router fallback tests 2026-04-20 16:10:45 -07:00
test_router_get_deployments.py Fix:add async_get_available_deployment_for_pass_through in code tests 2026-01-16 16:37:44 +05:30
test_router_max_parallel_requests.py Revert "Fix xdist test isolation: capture true defaults and poll instead of sleep" 2026-03-15 22:57:39 -07:00
test_router_pattern_matching.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_router_retries.py fix(tests): read CI_CD_DEFAULT_ANTHROPIC_MODEL env var instead of hardcoding model (#21781) 2026-02-21 10:46:49 -08:00
test_router_timeout.py Litellm fix update bedrock models (#24947) 2026-04-01 19:22:54 -07:00
test_router_utils.py test: test 2026-03-28 19:17:38 -07:00
test_router_with_fallbacks.py
test_router.py test(vcr): drop dead 'from respx import MockRouter' imports 2026-05-13 00:32:03 +00:00
test_rules.py
test_sagemaker_nova_integration.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_sagemaker.py Fix test_completion_sagemaker_messages_api retry flakiness 2026-03-15 13:57:50 -07:00
test_scheduler.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_secret_detect_hook.py
test_spend_calculate_endpoint.py
test_stream_chunk_builder.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_streaming.py [Fix] Tests: Replace deprecated openrouter/claude-3.7-sonnet with claude-sonnet-4.5 2026-05-04 17:51:50 -07:00
test_supabase_integration.py
test_team_config.py
test_text_completion.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_timeout.py Litellm fix update bedrock models (#24947) 2026-04-01 19:22:54 -07:00
test_together_ai.py
test_tpm_rpm_routing_v2.py fix: drain logging worker in test_router_caching_ttl to remove flake 2026-04-23 14:48:02 -07:00
test_traceloop.py
test_ui_sso_helper_utils.py
test_unit_test_caching.py style: black format test_unit_test_caching.py 2026-04-15 18:19:04 -07:00
test_update_spend.py fix(tests): skip remaining real prisma DB tests in CI and related test suites 2026-02-20 13:25:42 -03:00
test_validate_environment.py
test_wandb.py
user_cost.json
vertex_ai.jsonl
vertex_batch_completions.jsonl
vertex_key.json test: update to new vertex ai keys 2026-03-28 20:19:05 -07:00
whitelisted_bedrock_models.txt Litellm fix update bedrock models (#24947) 2026-04-01 19:22:54 -07:00