OpenAI Proxy Server

A simple, fast, and lightweight OpenAI-compatible server to call 100+ LLM APIs.

:::info We want to learn how we can make the proxy better! Meet the founders or join our discord :::

Usage

$ git clone https://github.com/BerriAI/litellm.git

$ cd ./litellm/openai-proxy

$ uvicorn main:app --host 0.0.0.0 --port 8000

Replace openai base

import openai 
openai.api_base = "http://0.0.0.0:8000" # proxy url
openai.api_key = "does-not-matter"
# call cohere
response = openai.ChatCompletion.create(
    model="command-nightly", 
    messages=[{"role":"user", "content":"Hey!"}],
    api_key="your-cohere-api-key", # enter your key here
)

# call bedrock 
response = openai.ChatCompletion.create(
    model = "bedrock/anthropic.claude-instant-v1",
    messages = [
        {
            "role": "user",
            "content": "Hey!"
        }
    ],
    aws_access_key_id="",
    aws_secret_access_key="",
    aws_region_name="us-west-2",
)

print(response)

See how to call Huggingface,Bedrock,TogetherAI,Anthropic, etc.

:::info Looking for the CLI tool/local proxy? It's here :::

Deploy on Google Cloud Run

Set LLM API Keys

Environment Variables

More info here

In the Google Cloud console, go to Cloud Run: Go to Cloud Run
Click on the litellm service <Image img={require('../img/cloud_run1.png')} />
Click Edit and Deploy New Revision <Image img={require('../img/cloud_run2.png')} />
Enter your Environment Variables Example OPENAI_API_KEY, ANTHROPIC_API_KEY <Image img={require('../img/cloud_run3.png')} />

2.2 KiB Raw Blame History