LiteLLM

---
schema: "https://insidethemachine.ai/schema/listing.schema.json"
slug: "litellm"
name: "LiteLLM"
category: "Model Routing"
stack_layer: "runtime"
homepage: "https://www.litellm.ai/"
repo: "https://github.com/BerriAI/litellm"
docs: "https://docs.litellm.ai/"
license: "See repo LICENSE (GitHub reports NOASSERTION: open-source core with a separately licensed enterprise directory)"
platforms: ["macos-apple-silicon", "linux", "windows"]
mac_studio_fit: "Primary"
stars: 61319
stars_as_of: "2026-10-15"
last_push: "2026-10-15"
gap_score: null
gap_score_status: "pending test on Mac Studio"
benchmarks_status: "pending test on Mac Studio"
verified_on: "2026-10-15"
sponsored: false
---

# LiteLLM

> One OpenAI-format API (SDK or proxy) in front of 100+ providers, including your local models.

**What:** One API in front of every model, local or cloud. **Why:** Send easy jobs to your Mac and hard ones to the cloud.

**Stack layer:** runtime (model → runtime → harness → memory → tools)

A Python SDK and proxy server (AI gateway) that exposes many LLM providers, including Ollama, behind an OpenAI-compatible API.

**Gap role:** Model routing and fallbacks: mix local and cloud models behind one endpoint so agents don't care where a call runs.

## Gap Score

null (pending test on Mac Studio)

## Prerequisites

- uv (brew install uv) or pip
- A backend, e.g. Ollama on localhost:11434

## Install

### macOS (Apple Silicon)

1. Run:

```bash
brew install uv
```

2. Run:

```bash
uv tool install 'litellm[proxy]'
```

### Linux

1. Run:

```bash
uv tool install 'litellm[proxy]'
```

### Windows

1. Run:

```bash
uv tool install "litellm[proxy]"
```

## Configure

1. maps a local Ollama model to a proxy alias (format from LiteLLM's Ollama docs):

```bash
cat > litellm.yaml <<'EOF'
model_list:
  - model_name: "local-llama"
    litellm_params:
      model: "ollama_chat/llama3.1"
EOF
```

2. proxy on http://0.0.0.0:4000:

```bash
litellm --config litellm.yaml
```

## Verify

1. Expect: OpenAI-format JSON response; add an Authorization header if you set a master key

```bash
curl -s http://0.0.0.0:4000/chat/completions -H 'Content-Type: application/json' -d '{"model":"local-llama","messages":[{"role":"user","content":"hi"}]}'
```

## Rollback

1. Run:

```bash
uv tool uninstall litellm
```

2. Run:

```bash
rm litellm.yaml
```

## One-command install

Status: docs-checked (Not run: commands checked against the official docs only). The wrapper scripts are generated from the site data. Every .sh passed bash -n and shellcheck and ran in plan mode on Linux (Debian 13). Every .ps1 passed the PowerShell parser and chat.ps1 ran in plan, verify and rollback mode under PowerShell 7 on Linux with Windows calls mocked. Not run on macOS or real Windows.

```bash
curl -fsSL https://insidethemachine.ai/install/litellm.sh | bash            # install, then check
curl -fsSL https://insidethemachine.ai/install/litellm.sh | bash -s -- plan    # print only
curl -fsSL https://insidethemachine.ai/install/litellm.sh | bash -s -- rollback --yes
```

Details and Windows: https://insidethemachine.ai/install/litellm.md

## Caveats

- Requires `ollama pull llama3.1` (or edit the model) before the verify step.

## Benchmarks

Status: pending test on Mac Studio. Rig: Mac Studio, Apple Silicon, 128GB unified memory. No results published yet.

## Works with

- ollama (https://insidethemachine.ai/setups/ollama.md)
- mlx-lm (https://insidethemachine.ai/setups/mlx-lm.md)
- llama-cpp (https://insidethemachine.ai/setups/llama-cpp.md)
- lm-studio (https://insidethemachine.ai/setups/lm-studio.md)
- aider (https://insidethemachine.ai/setups/aider.md)
- open-webui (https://insidethemachine.ai/setups/open-webui.md)

## Sources

- https://github.com/BerriAI/litellm
- https://docs.litellm.ai/docs/providers/ollama