ai-workspace-services/litellm
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

update benchmarks

Browse Source
This commit is contained in:
Ishaan Jaff 2025-01-14 10:45:28 -08:00
parent 00f50bc201
commit eb2770fee2
1 changed files with 5 additions and 3 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

8
docs/my-website/docs/benchmarks.md
View File

@ -20,11 +20,13 @@ model_list:
## 1 Instance LiteLLM Proxy
In these tests the median latency of directly calling the fake-openai-endpoint is 60ms.
| Metric | Litellm Proxy (1 Instance) |
|--------|------------------------|
| Median Latency (ms) | 110 |
| RPS | 250 |
| RPS | 500 |
| Median Latency (ms) | 100 |
| Latency overhead added by LiteLLM Proxy | 40ms |
<!-- <Image img={require('../img/1_instance_proxy.png')} /> -->
@ -33,7 +35,7 @@ model_list:
<Image img={require('../img/instances_vs_rps.png')} /> -->
#### Key Findings
- Single instance: 250 RPS @ 100ms latency
- Single instance: 500 RPS @ 100ms latency
- 4 LiteLLM instances: 1000 RPS @ 100ms latency
### 2 Instances