🐳 Docker, Deploying LiteLLM Proxy | liteLLM (2024)

You can find the Dockerfile to build litellm proxy here

Quick Start​

model_list:
 - model_name: azure-gpt-3.5
 litellm_params:
 model: azure/<your-azure-model-deployment>
 api_base: os.environ/AZURE_API_BASE
 api_key: os.environ/AZURE_API_KEY
 api_version: "2023-07-01-preview"

docker run \
 -v $(pwd)/litellm_config.yaml:/app/config.yaml \
 -e AZURE_API_KEY=d6*********** \
 -e AZURE_API_BASE=https://openai-***********/ \
 -p 4000:4000 \
 ghcr.io/berriai/litellm:main-latest \
 --config /app/config.yaml --detailed_debug

curl --location 'http://0.0.0.0:4000/chat/completions' \
 --header 'Content-Type: application/json' \
 --data '{
 "model": "azure-gpt-3.5",
 "messages": [
 {
 "role": "user",
 "content": "what llm are you"
 }
 ]
}'

That's it ! That's the quick start to deploy litellm

Options to deploy LiteLLM​

Deploy with Database​

Docker, Kubernetes, Helm Chart​

Requirements:

• Need a postgres database (e.g. Supabase, Neon, etc) Set DATABASE_URL=postgresql://<user>:<password>@<host>:<port>/<dbname> in your env
• Set a LITELLM_MASTER_KEY, this is your Proxy Admin key - you can use this to create other keys (🚨 must start with sk-)

docker pull ghcr.io/berriai/litellm-database:main-latest

docker run \
 -v $(pwd)/litellm_config.yaml:/app/config.yaml \
 -e LITELLM_MASTER_KEY=sk-1234 \
 -e DATABASE_URL=postgresql://<user>:<password>@<host>:<port>/<dbname> \
 -e AZURE_API_KEY=d6*********** \
 -e AZURE_API_BASE=https://openai-***********/ \
 -p 4000:4000 \
 ghcr.io/berriai/litellm-database:main-latest \
 --config /app/config.yaml --detailed_debug

LiteLLM container + Redis​

Use Redis when you need litellm to load balance across multiple litellm containers

The only change required is setting Redis on your config.yamlLiteLLM Proxy supports sharing rpm/tpm shared across multiple litellm instances, pass redis_host, redis_password and redis_port to enable this. (LiteLLM will use Redis to track rpm/tpm usage )

model_list:
 - model_name: gpt-3.5-turbo
 litellm_params:
 model: azure/<your-deployment-name>
 api_base: <your-azure-endpoint>
 api_key: <your-azure-api-key>
 rpm: 6 # Rate limit for this deployment: in requests per minute (rpm)
 - model_name: gpt-3.5-turbo
 litellm_params:
 model: azure/gpt-turbo-small-ca
 api_base: https://my-endpoint-canada-berri992.openai.azure.com/
 api_key: <your-azure-api-key>
 rpm: 6
router_settings:
 redis_host: <your redis host>
 redis_password: <your redis password>
 redis_port: 1992

Start docker container with config

docker run ghcr.io/berriai/litellm:main-latest --config your_config.yaml

LiteLLM Database container + PostgresDB + Redis​

The only change required is setting Redis on your config.yamlLiteLLM Proxy supports sharing rpm/tpm shared across multiple litellm instances, pass redis_host, redis_password and redis_port to enable this. (LiteLLM will use Redis to track rpm/tpm usage )

model_list:
 - model_name: gpt-3.5-turbo
 litellm_params:
 model: azure/<your-deployment-name>
 api_base: <your-azure-endpoint>
 api_key: <your-azure-api-key>
 rpm: 6 # Rate limit for this deployment: in requests per minute (rpm)
 - model_name: gpt-3.5-turbo
 litellm_params:
 model: azure/gpt-turbo-small-ca
 api_base: https://my-endpoint-canada-berri992.openai.azure.com/
 api_key: <your-azure-api-key>
 rpm: 6
router_settings:
 redis_host: <your redis host>
 redis_password: <your redis password>
 redis_port: 1992

Start litellm-databasedocker container with config

docker run --name litellm-proxy \
-e DATABASE_URL=postgresql://<user>:<password>@<host>:<port>/<dbname> \
-p 4000:4000 \
ghcr.io/berriai/litellm-database:main-latest --config your_config.yaml

Advanced Deployment Settings​

Customization of the server root path​

Customize the root path to eliminate the need for employing multiple DNS configurations during deployment.

👉 Set SERVER_ROOT_PATH in your .env and this will be set as your server root path

Setting SSL Certification​

Use this, If you need to set ssl certificates for your on prem litellm proxy

Pass ssl_keyfile_path (Path to the SSL keyfile) and ssl_certfile_path (Path to the SSL certfile) when starting litellm proxy

docker run ghcr.io/berriai/litellm:main-latest \
 --ssl_keyfile_path ssl_test/keyfile.key \
 --ssl_certfile_path ssl_test/certfile.crt

Provide an ssl certificate when starting litellm proxy server

Platform-specific Guide​

eksctl create cluster --name=litellm-cluster --region=us-west-2 --node-type=t2.small

kubectl create configmap litellm-config --from-file=proxy_config.yaml

kubectl apply -f kub.yaml

kubectl apply -f service.yaml

# service/litellm-service created

kubectl get services

# litellm-service LoadBalancer 10.100.6.31 a472dc7c273fd47fd******.us-west-2.elb.amazonaws.com 4000:30374/TCP 63m

Extras​

Run with docker compose​

Step 1

• (Recommended) Use the example file docker-compose.yml given in the project root. e.g. https://github.com/BerriAI/litellm/blob/main/docker-compose.yml

Here's an example docker-compose.yml file

version: "3.9"
services:
 litellm:
 build:
 context: .
 args:
 target: runtime
 image: ghcr.io/berriai/litellm:main-latest
 ports:
 - "4000:4000" # Map the container port to the host, change the host port if necessary
 volumes:
 - ./litellm-config.yaml:/app/config.yaml # Mount the local configuration file
 # You can change the port or number of workers as per your requirements or pass any new supported CLI augument. Make sure the port passed here matches with the container port defined above in `ports` value
 command: [ "--config", "/app/config.yaml", "--port", "4000", "--num_workers", "8" ]

# ...rest of your docker-compose config if any

Step 2

Create a litellm-config.yaml file with your LiteLLM config relative to your docker-compose.yml file.

Check the config doc here

Step 3

Run the command docker-compose up or docker compose up as per your docker installation.

Use -d flag to run the container in detached mode (background) e.g. docker compose up -d

Your LiteLLM container should be running now on the defined port e.g. 4000.