MLX LM

---
schema: "https://insidethemachine.ai/schema/listing.schema.json"
slug: "mlx-lm"
name: "MLX LM"
category: "Runtimes / Inference"
stack_layer: "runtime"
homepage: "https://github.com/ml-explore/mlx-lm"
repo: "https://github.com/ml-explore/mlx-lm"
docs: "https://github.com/ml-explore/mlx-lm/blob/main/mlx_lm/SERVER.md"
license: "MIT"
platforms: ["macos-apple-silicon"]
mac_studio_fit: "Primary"
stars: 7310
stars_as_of: "2026-10-15"
last_push: "2026-10-10"
gap_score: null
gap_score_status: "pending test on Mac Studio"
benchmarks_status: "pending test on Mac Studio"
verified_on: "2026-10-15"
sponsored: false
---

# MLX LM

> Apple's MLX-native way to generate, chat, fine-tune and serve LLMs on Apple silicon.

**What:** Runs LLMs natively on Apple silicon. **Why:** Built for the chip in your Mac, so you get more model per GB.

**Stack layer:** runtime (model → runtime → harness → memory → tools)

A Python package (mlx-lm) for running and fine-tuning large language models with Apple's MLX framework. Ships CLIs for generate, chat, convert/quantize, and an OpenAI-like HTTP server.

**Gap role:** Raw local inference speed and memory efficiency on Apple silicon; quantization (mlx_lm.convert -q) and KV-cache quantization (--kv-bits) let bigger models fit in unified memory.

## Gap Score

null (pending test on Mac Studio)

## Prerequisites

- Apple silicon Mac
- Python 3 with pip (a virtualenv is recommended)
- Some features (e.g. wired-memory tuning) require macOS 15+ per the README

## Install

### macOS (Apple Silicon)

1. isolated environment (recommended, not required):

```bash
python3 -m venv ~/.venvs/mlx && source ~/.venvs/mlx/bin/activate
```

2. Run:

```bash
pip install mlx-lm
```

3. Run:

```bash
# or with conda: conda install -c conda-forge mlx-lm
```

## Configure

1. starts an OpenAI-like server on localhost:8080; model downloads from Hugging Face on first run:

```bash
mlx_lm.server --model mlx-community/Mistral-7B-Instruct-v0.3-4bit
```

2. quantized KV cache for long contexts:

```bash
mlx_lm.server --model <hf_repo_or_path> --kv-bits 4 --prefill-step-size 512
```

## Verify

1. Expect: prints generated text and tokens-per-second stats

```bash
mlx_lm.generate --model mlx-community/Mistral-7B-Instruct-v0.3-4bit --prompt "hello"
```

2. Expect: JSON response with a choices[0].message.content field (server must be running)

```bash
curl -s http://localhost:8080/v1/chat/completions -H 'Content-Type: application/json' -d '{"messages":[{"role":"user","content":"Say hi"}]}'
```

## Rollback

1. Run:

```bash
pip uninstall -y mlx-lm
```

2. only if you created this venv:

```bash
rm -rf ~/.venvs/mlx
```

3. Run:

```bash
# downloaded models live in the Hugging Face cache (~/.cache/huggingface/hub); delete specific model folders to free space
```

## One-command install

Status: docs-checked (Not run: commands checked against the official docs only). The wrapper scripts are generated from the site data. Every .sh passed bash -n and shellcheck and ran in plan mode on Linux (Debian 13). Every .ps1 passed the PowerShell parser and chat.ps1 ran in plan, verify and rollback mode under PowerShell 7 on Linux with Windows calls mocked. Not run on macOS or real Windows.

```bash
curl -fsSL https://insidethemachine.ai/install/mlx-lm.sh | bash            # install, then check
curl -fsSL https://insidethemachine.ai/install/mlx-lm.sh | bash -s -- plan    # print only
curl -fsSL https://insidethemachine.ai/install/mlx-lm.sh | bash -s -- rollback --yes
```

Details and Windows: https://insidethemachine.ai/install/mlx-lm.md

## Caveats

- mlx_lm.server is documented as not recommended for production (basic security checks only).
- Port 8080 is also the default for llama.cpp's server and Open WebUI's pip install; run them on different ports.

## Benchmarks

Status: pending test on Mac Studio. Rig: Mac Studio, Apple Silicon, 128GB unified memory. No results published yet.

## Works with

- open-webui (https://insidethemachine.ai/setups/open-webui.md)
- litellm (https://insidethemachine.ai/setups/litellm.md)
- aider (https://insidethemachine.ai/setups/aider.md)

## Sources

- https://github.com/ml-explore/mlx-lm
- https://github.com/ml-explore/mlx-lm/blob/main/mlx_lm/SERVER.md