ai-workspace-services/litellm
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
2a5dfcd5bc
litellm/tests/test_litellm
History
Yassin Kortam 10bd7406e0
feat: add guardrail violation span attributes and fix missing spans on pre-call blocks (#28364) 
- Fix missing guardrail child spans when a pre-call guardrail blocks the request before reaching the LLM provider; `async_post_call_failure_hook` now calls `_emit_guardrail_spans_from_request_data` to emit spans from `request_data["metadata"]` regardless of whether `_handle_failure` already fired
- Add `guardrail_status`, `guardrail_action`, and `guardrail_violation_categories` as queryable top-level OTEL span attributes so trace backends can filter/group by violation type without parsing the redacted `guardrail_response` blob
- Introduce `_emit_guardrail_spans_from_request_data` helper that constructs minimal kwargs from `request_data["metadata"]` and routes through `_create_guardrail_span`, sharing the same dedupe state to prevent double-emitting when both failure hooks fire
- Extend `BedrockGuardrail` with `_build_tracing_detail` and `_extract_violation_category_names` which flatten BLOCKED assessments into human-readable category labels (topic names, content-filter types, PII entity types, named regex names) before redaction, and surface Bedrock's raw `action` field via `tracing_detail`
- Security: violation category extraction deliberately omits `customWords.match` and unnamed regex `match` values because those fields carry the user-submitted content that triggered the rule; only operator-defined `name`/`type` labels are emitted
- Add `violation_categories` and `guardrail_action` fields to `StandardLoggingGuardrailInformation` and `GuardrailTracingDetail` TypedDicts to carry the pre-redaction metadata through the logging pipeline
- Add comprehensive test suite covering: guardrail span creation on failure, dedupe between `_handle_failure` and `async_post_call_failure_hook`, per-span status attributes for multi-guardrail sequences, Bedrock category extraction for all policy types, security leak prevention, and end-to-end `CustomGuardrail` violation path

Co-authored-by: Yassin Kortam <yassinkortam@g.ucla.edu>
2026-05-21 15:49:42 -07:00
..
a2a_protocol style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
anthropic_interface/exceptions style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
caching fix(caching): replay openai/responses bridge cache hits as chat streams (#28158) 2026-05-18 16:27:06 -07:00
completion_extras Litellm oss staging 04 21 2026 2 (#26569) 2026-05-20 21:25:19 -07:00
containers fix(router): use forwarded model_id for native Azure container IDs (#27921) 2026-05-20 12:32:34 -07:00
enterprise fix(proxy): normalize batch file IDs before ManagedObjectTable write (#28339) 2026-05-20 12:13:56 -07:00
expected_fine_tuning_api refactor: refactor testing 2026-03-28 18:39:32 -07:00
expected_responses_api_request
experimental_mcp_client style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
google_genai fix(gemini): normalize response_schema on native generateContent (#27775) 2026-05-12 23:26:34 -07:00
images fix(image_edit): read vertex_project/location from litellm_params in Imagen get_complete_url 2026-04-21 15:03:40 +05:30
integrations feat: add guardrail violation span attributes and fix missing spans on pre-call blocks (#28364) 2026-05-21 15:49:42 -07:00
interactions fix(interactions): never drop streamed text deltas; always emit terminal completion (#28394) 2026-05-20 16:41:40 -07:00
litellm_core_utils fix(logging): recalculate cost after router retry failures (#28476) 2026-05-21 14:07:05 -07:00
llms fix(vertex_gemma): strip context_management from request body (#28438) 2026-05-21 23:10:33 +05:30
ocr chore: reject bare str at file-input sinks to prevent local-file read (#27762) 2026-05-12 16:40:07 -07:00
passthrough refactor: trim explanatory comments from streaming-flush fix 2026-04-30 02:39:28 +00:00
proxy feat: add guardrail violation span attributes and fix missing spans on pre-call blocks (#28364) 2026-05-21 15:49:42 -07:00
responses Litellm oss staging 04 21 2026 2 (#26569) 2026-05-20 21:25:19 -07:00
router_strategy feat: selectively apply routing strategy according to model name 2026-05-04 13:27:32 -07:00
router_utils Gemini managed agents support (#28270) 2026-05-19 16:02:03 -07:00
secret_managers Implement normalize_nonempty_secret_str function to trim whitespace from secrets and treat empty values as unset. Update proxy_server to use this function for Grafana credentials. Enhance tests to validate the new normalization behavior. 2026-05-04 18:17:31 +00:00
test_router
types Merge pull request #25256 from BerriAI/litellm_ishaan_april6 2026-04-17 16:26:45 -07:00
vector_stores style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
__init__.py
conftest.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
log.txt
readme.md
test_a2a_registry_lookup.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_acompletion_session_reuse_e2e.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_add_deployment_no_master_key.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_aembedding_session_reuse_e2e.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_anthropic_beta_headers_filtering.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_anthropic_skills_transformation.py fix: encode upstream URL path identifiers 2026-04-29 22:02:39 -07:00
test_azure_video_router.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_bedrock_anthropic_1hr_cache_pricing.py fix(bedrock): add 1-hour cache write tier for Claude 4.5/4.6/4.7 (Global, US) 2026-04-29 19:21:57 +00:00
test_chat_ui_responses_session.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_claude_haiku_4_5_config.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_claude_opus_4_6_config.py Revert "Merge pull request #24164 from dongyu-turo/feat/update-bedrock-claude-price-above-200k" 2026-04-25 15:04:05 -03:00
test_claude_sonnet_4_6_config.py fix(bedrock-mantle): use /anthropic/v1/messages path for Mantle endpo… (#27976) 2026-05-15 13:31:59 -07:00
test_completion_timeout_resolution.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_compression.py Prompt Compression - add it to the proxy (#25729) 2026-04-20 15:08:00 -07:00
test_constants.py Fix test 2026-03-27 21:21:43 +05:30
test_container_router.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_cost_calculation_log_level.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_cost_calculator.py Litellm oss staging 1 (#28337) 2026-05-20 17:27:03 -07:00
test_count_tokens_public_api.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_dashscope_image_generation.py fix(dashscope): fail fast on image generation API errors 2026-04-23 18:41:01 +05:30
test_deepseek_model_metadata.py Revert "Merge pull request #16590 from Chesars/refactor/remove-backup-file-dry-principle" 2026-04-25 17:10:41 -03:00
test_dockerfile_non_root.py fix(docker.non_root): use numeric UID 65534 for K8s runAsNonRoot (#26268) 2026-04-22 18:00:04 -07:00
test_eager_tiktoken_load.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_exception_exports.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_exception_header_preservation.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_exception_mapping_request_attribute.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_filter_out_litellm_params.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_get_blog_posts.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_gpt_image_cost_calculator.py Add gpt-image-2 support (#26644) (#26705) 2026-04-28 20:10:42 -07:00
test_groq_streaming_encoding.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_guardrail_exception_status_codes.py Litellm oss staging (#28161) 2026-05-18 16:27:44 -07:00
test_lazy_imports.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_litellm_params_reserved_keys.py
test_logging.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_lowest_latency_zero_tokens.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_main_module_header.py Add main module header comment 2026-05-06 00:26:17 +00:00
test_main.py Match litellm.completion supported model parameters with proxy model info (#27720) 2026-05-12 08:25:01 -07:00
test_model_cost_aliases.py
test_model_param_helper.py style: black format test_model_param_helper.py 2026-04-15 18:18:52 -07:00
test_model_response_normalization.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_nested_drop_params.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_openai_embedding_encoding_format_default.py fix(embeddings): allow omitting encoding_format via env sentinel none 2026-05-01 23:03:13 +05:30
test_project_alias_tracking.py feat(proxy): add project_alias tracking through callback metadata pipeline 2026-03-23 10:44:17 -07:00
test_project_tags_pydantic.py
test_redact_string_in_error_paths.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_redis.py Litellm oss staging (#28161) 2026-05-18 16:27:44 -07:00
test_register_model_custom_pricing.py
test_responses_api_bridge_non_stream.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_responses_id_security.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_retrieve_batch_bedrock_dispatch.py feat(batch-job): bedrock batch model invocation job retrieval (#26834) 2026-05-11 13:22:26 -07:00
test_router_google_genai.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_router_model_cost_isolation.py Litellm oss staging 04 21 2026 2 (#26569) 2026-05-20 21:25:19 -07:00
test_router_order_fallback.py fix(router): enable order fallback for wildcard model groups 2026-04-15 17:48:25 +05:30
test_router_per_deployment_num_retries.py
test_router_redis_init.py
test_router_retry_non_retryable_errors.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_router_silent_experiment.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_router_weighted_failover.py feat: add weighted-routing failover (#27980) 2026-05-15 17:28:54 +00:00
test_router.py Litellm oss staging 1 (#28337) 2026-05-20 17:27:03 -07:00
test_secret_redaction.py Unify secret redaction patterns 2026-05-01 17:24:21 +00:00
test_service_logger.py
test_setup_wizard.py test: test 2026-03-28 19:17:38 -07:00
test_shared_session_integration.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_ssl_verify_unit.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_stream_chunk_builder_annotations.py
test_streaming_connection_cleanup.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_system_message_format_bug.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_utils_module_docstring.py Add utils module docstring 2026-05-06 00:42:49 +00:00
test_utils.py Litellm oss staging 04 21 2026 2 (#26569) 2026-05-20 21:25:19 -07:00
test_uuid_helper.py
test_vcr_safe_body_matcher.py test: add 24hr Redis-backed VCR cache to additional test suites (#27159) 2026-05-05 15:13:31 -07:00
test_video_generation.py feat(gemini): Veo Lite pricing, size→resolution, usage video_resolution for cost tiers 2026-04-08 19:50:50 +05:30
test_xai_grok_4_3_model_metadata.py feat(xai): add grok-4.3 and grok-4.3-latest to model_prices_and_conte… (#27154) 2026-05-07 09:06:56 -07:00
test_xai_responses_auto_routing.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00

readme.md

Testing for litellm/

This directory 1:1 maps the the litellm/ directory, and can only contain mocked tests.

The point of this is to:

  1. Increase test coverage of litellm/
  2. Make it easy for contributors to add tests for the litellm/ package and easily run tests without needing LLM API keys.

File name conventions

  • litellm/proxy/test_caching_routes.py maps to litellm/proxy/caching_routes.py
  • test_<filename>.py maps to litellm/<filename>.py