llama.cpp

---
schema: "https://insidethemachine.ai/schema/listing.schema.json"
slug: "llama-cpp"
name: "llama.cpp"
category: "Runtimes / Inference"
stack_layer: "runtime"
homepage: "https://llama.app"
repo: "https://github.com/ggml-org/llama.cpp"
docs: "https://github.com/ggml-org/llama.cpp/blob/master/docs/build.md"
license: "MIT"
platforms: ["macos-apple-silicon", "linux", "windows"]
mac_studio_fit: "Primary"
stars: 131267
stars_as_of: "2026-10-15"
last_push: "2026-10-15"
gap_score: null
gap_score_status: "pending test on Mac Studio"
benchmarks_status: "pending test on Mac Studio"
verified_on: "2026-10-15"
sponsored: false
---

# llama.cpp

> The gold-standard C/C++ inference engine for GGUF models, with Metal as a first-class backend.

**What:** The engine behind most local AI. **Why:** Full control over speed, context and quantization.

**Stack layer:** runtime (model → runtime → harness → memory → tools)

LLM inference in C/C++. Runs GGUF models from Hugging Face, includes a CLI and an OpenAI-compatible server with a built-in web UI. README: 'Apple silicon is a first-class citizen - optimized via ARM NEON, Accelerate and Metal frameworks'.

**Gap role:** Maximum control over quantization, context and sampling; widest GGUF model coverage.

## Gap Score

null (pending test on Mac Studio)

## Prerequisites

- macOS/Linux shell or Windows PowerShell

## Install

### macOS (Apple Silicon)

1. official installer from the README:

```bash
curl -LsSf https://llama.app/install.sh | sh
```

2. Run:

```bash
# alternative: brew install llama.cpp   (Homebrew formula is at 0.6.0 as of 2026-10-15)
```

### Linux

1. Run:

```bash
curl -LsSf https://llama.app/install.sh | sh
```

### Windows

1. PowerShell:

```bash
irm https://llama.app/install.ps1 | iex
```

## Configure

1. download from Hugging Face and chat (README example):

```bash
llama cli -hf ggml-org/Qwen3.5-0.8B-GGUF
```

2. OpenAI-compatible API server + web UI:

```bash
llama serve -hf ggml-org/Qwen3.5-0.8B-GGUF
```

## Verify

1. Expect: path to the llama binary (Homebrew/older builds may expose llama-cli / llama-server instead)

```bash
command -v llama
```

2. Expect: JSON listing the loaded model (server running; 8080 is the server's usual default, check startup log)

```bash
curl -s http://localhost:8080/v1/models
```

## Rollback

1. if installed via Homebrew:

```bash
brew uninstall llama.cpp
```

2. Run:

```bash
# script install: follow uninstall instructions at https://llama.app
```

3. Run:

```bash
# -hf downloads are cached (macOS default ~/Library/Caches/llama.cpp); delete to free space
```

## One-command install

Status: docs-checked (Not run: commands checked against the official docs only). The wrapper scripts are generated from the site data. Every .sh passed bash -n and shellcheck and ran in plan mode on Linux (Debian 13). Every .ps1 passed the PowerShell parser and chat.ps1 ran in plan, verify and rollback mode under PowerShell 7 on Linux with Windows calls mocked. Not run on macOS or real Windows.

```bash
curl -fsSL https://insidethemachine.ai/install/llama-cpp.sh | bash            # install, then check
curl -fsSL https://insidethemachine.ai/install/llama-cpp.sh | bash -s -- plan    # print only
curl -fsSL https://insidethemachine.ai/install/llama-cpp.sh | bash -s -- rollback --yes
```

Details and Windows: https://insidethemachine.ai/install/llama-cpp.md

## Caveats

- CLI naming changed across versions (llama cli / llama serve vs llama-cli / llama-server); use whichever your install provides.

## Benchmarks

Status: pending test on Mac Studio. Rig: Mac Studio, Apple Silicon, 128GB unified memory. No results published yet.

## Works with

- open-webui (https://insidethemachine.ai/setups/open-webui.md)
- litellm (https://insidethemachine.ai/setups/litellm.md)
- aider (https://insidethemachine.ai/setups/aider.md)

## Sources

- https://github.com/ggml-org/llama.cpp
- https://formulae.brew.sh/formula/llama.cpp
- https://www.youtube.com/watch?v=pWu3cNd0Qyo