Local AI in plain words.

# Start here: local AI in plain words

URL: https://insidethemachine.ai/start-here

## Three parts
- Model: the AI itself. One big file you download. It holds what the AI learned.
- Ollama: the engine. It loads the model and runs it on your computer. Free.
- App: the window you talk to (Open WebUI, LM Studio, OpenCode, Hermes). It sends your words to Ollama and shows the answer.
All three run on your computer. Nothing goes to the internet after the download.

## Memory decides which models fit
The whole model must fit in memory while it runs. On a Mac about 75% of memory is usable for models. On a PC with a graphics card, the card's own memory (VRAM) counts, not the computer's RAM.
- 8 GB computer (~6 GB usable): Qwen3.5 4B, needs ~5.8 GB, about 23% as smart as Claude Opus 5.5
- 16 GB computer (~12 GB usable): K2 Horizon 7B, needs ~8.6 GB, about 36% as smart as Claude Opus 5.5
- 32 GB computer (~24 GB usable): Qwen3.8 27B, needs ~22.7 GB, about 59% as smart as Claude Opus 5.5
Find your memory: Mac: Apple menu > About This Mac. Windows: Settings > System > About > Installed RAM. Graphics card: Task Manager > Performance > GPU > Dedicated GPU memory.

## What "27B" means
B = billion. A 27B model has 27 billion numbers it learned. More numbers: smarter, bigger file, slower. Example: Qwen3.8 27B is a 27B model; its Q4 file is 18 GB and it needs ~22.7 GB of memory with room for the chat.
"36B-A4B" = 36 billion numbers in total, only 4 billion used for each word. It needs the memory of a 36B model but runs at the speed of a 4B model.

## Speed
Speed is tokens per second (tok/s). People read at about 5 tok/s.
- Under 10 tok/s: about reading speed or slower. Example: llama3.2 3B on a Raspberry Pi 5 (CPU only): 5.6 tok/s.
- 10 to 40 tok/s: comfortable.
- 40+ tok/s: faster than you can read. Example: gpt-oss-20b on an M5 Pro Mac: 76.6 tok/s.
Memory bandwidth (GB/s) sets speed. The first answer is slow while the model loads.

## How smart
We compare each model with Claude Opus 5.5, the best paid AI, on the Artificial Analysis Intelligence Index. 100% = as smart as Claude Opus 5.5.

## Is it safe?
- The install command downloads Ollama's official script from ollama.com and runs it. Only run commands from sites you trust; you can open the script URL in a browser to read it first.
- Ollama only listens on your own computer (127.0.0.1:11434) unless you change it.
- Test: after the download, turn off Wi-Fi. It still answers.
- Models come from the Ollama library or Hugging Face. Check the licence before business use.

## Words
- Token: A piece of a word. 100 tokens is about 75 English words.
- Context: How much text the model can read at once: your chat, files and instructions. More context needs more memory.
- Q4 (4-bit): A smaller copy of a model. It keeps most of the smarts and needs less than a third of the space. The sizes on this site are Q4.
- Open weights: The maker lets you download the model file. Check its licence before you use it for business.
- Harness: Another word for the app around the model. It can read files, use tools and remember things.
- GPU: The part of the computer that does the maths fast. On a Mac it shares the normal memory. On a PC it is the graphics card.

Next: pick a goal at https://insidethemachine.ai/#goals

## Recipes (do one next)
- A private ChatGPT (needs 8 GB+, best 16 GB+, ~10 min): https://insidethemachine.ai/do/chat.md
- A coding helper (needs 16 GB+, best 32 GB+, ~15 min): https://insidethemachine.ai/do/code.md
- Chat with your files (needs 16 GB+, best 32 GB+, ~20 min): https://insidethemachine.ai/do/docs.md
- Speech to text (needs 8 GB+, best 16 GB+, ~10 min): https://insidethemachine.ai/do/voice.md
- Make images (needs 16 GB+, best 32 GB+, ~20 min): https://insidethemachine.ai/do/images.md
- An agent that does tasks (needs 32 GB+, best 64 GB+, ~30 min): https://insidethemachine.ai/do/agent.md

## Quick path for an agent setting up this machine
1. Read total RAM (macOS: `sysctl -n hw.memsize`, bytes) or GPU VRAM (`nvidia-smi --query-gpu=memory.total --format=csv`).
2. Pick the model for that memory size: https://insidethemachine.ai/models.md (table "Picks by RAM tier").
3. Pick the recipe for the user's goal from the list above and run its steps in order; check each "Expect" line before going on.
4. If a step fails, use that recipe's "If something goes wrong" list, then its Rollback.
5. For a self-improving agent setup (AGENTS.md, SOUL.md, memory, routing), continue with the full runbook below (also at https://insidethemachine.ai/start.md).

More: models https://insidethemachine.ai/models.md · machines https://insidethemachine.ai/machines.md · apps https://insidethemachine.ai/harnesses.md · catalog https://insidethemachine.ai/llms.txt · JSON https://insidethemachine.ai/start-here.json

---

---
title: "Zero to working local AI"
slug: start
kind: runbook
verified_on: 2026-10-15
status: "Commands verified against official docs on 2026-10-15; not yet run end-to-end on the Mac Studio test rig unless marked mac_studio_tested: true."
platform_priority: ["macos-apple-silicon", "linux", "windows"]
steps: 5
json: https://insidethemachine.ai/start.json
catalog: https://insidethemachine.ai/api/index.json
---

# Zero to working local AI

> Executable runbook. Run steps in order; each has goal, commands per platform, verify (with expected output) and rollback. Commands verified against official docs on 2026-10-15; not yet run end-to-end on the Mac Studio test rig unless marked mac_studio_tested: true.

## Bootstrap prompt

Paste this into your agent once step 3 works:

```text
You are my local AI setup agent. Work only inside ~/itm-agent. Ask before deleting anything, spending money, or sending anything outside this machine.

Context: a local model server is already running (Ollama at http://localhost:11434/v1 or LM Studio at http://localhost:1234/v1).

1. Read https://insidethemachine.ai/start.md and https://insidethemachine.ai/llms.txt. Use them as your runbook and catalog.
2. My goal: <describe what you want, in plain English>.
3. Split that goal into workflows (for example research, writing, files, documents, memory). Create ~/itm-agent with AGENTS.md (goal, workflows, rules), memory/MEMORY.md, and one folder per workflow, following step 4 of start.md.
4. For each workflow, pick the tools it needs from the catalog. For each tool, fetch https://insidethemachine.ai/setups/<slug>.md, run its install and verify steps, and write the result to memory/MEMORY.md. If a verify step fails, run that listing's rollback and tell me.
5. Set up LiteLLM routing (step 5): "fast" = my everyday local model, "smart" = the biggest model my RAM allows. Use smart only for hard tasks.
6. Finish with a short report: what you installed, what passed, what failed, and what you'd upgrade next. Note that official Gap Scores are still pending test on Mac Studio.
```

## Step 1: Check your machine

**Goal:** Know total RAM (Apple Silicon: unified memory) or GPU VRAM, and pick a model size from the fit chart.

### Fit chart

Rule of thumb (estimate, not measured): comfortable if the model file is at most 50% of RAM/VRAM, tight up to 75%, otherwise no. Context window and the OS need the rest. Sizes: https://ollama.com/library (tags pages, download sizes as listed 2026-10-15)

| RAM | qwen3.5:4b (3.3 GB) | gemma4 (6.6 GB) | gemma4:26b (16 GB) | qwen3-coder:30b (19 GB) | qwen3.5:122b (81 GB) | pick |
|---|---|---|---|---|---|---|
| 8 GB | comfortable | no | no | no | no | qwen3.5:4b |
| 16 GB | comfortable | comfortable | no | no | no | gemma4 |
| 32 GB | comfortable | comfortable | comfortable | tight | no | gemma4:26b |
| 64 GB | comfortable | comfortable | comfortable | comfortable | no | qwen3-coder:30b |
| 128 GB | comfortable | comfortable | comfortable | comfortable | tight | qwen3-coder:30b (fast) + qwen3.5:122b (smart) |

### Commands: macOS (Apple Silicon)

1. Run:

```bash
sysctl -n hw.memsize | awk '{printf "%.0f GB unified memory\n", $1/1073741824}'
```

2. Run:

```bash
system_profiler SPHardwareDataType | grep -E 'Chip|Memory'
```

### Commands: Linux

1. Run:

```bash
free -g | awk '/Mem:/{print $2" GB RAM"}'
```

2. Run:

```bash
nvidia-smi --query-gpu=name,memory.total --format=csv   # NVIDIA GPUs only
```

### Commands: Windows

1. Run:

```bash
(Get-CimInstance Win32_ComputerSystem).TotalPhysicalMemory/1GB   # PowerShell
```

### Verify

1. Expect: prints a byte count; 128GB Mac Studio ≈ 137438953472

```bash
sysctl -n hw.memsize
```

### Rollback

1. Run:

```bash
# read-only step; nothing to undo
```

Guides: Machines: what to buy or what yours can run -> https://insidethemachine.ai/machines.md (JSON: https://insidethemachine.ai/machines.json); Models vs memory: what fits your RAM -> https://insidethemachine.ai/models.md (JSON: https://insidethemachine.ai/models.json)

Related listings: https://insidethemachine.ai/setups/mlx-lm.md, https://insidethemachine.ai/setups/llama-cpp.md

## Step 2: Pick your tool

**Goal:** A local model server is running and answers a prompt.

### Commands: macOS (Apple Silicon)

1. Option A (recommended for agents): Ollama:

```bash
brew install ollama   # or: curl -fsSL https://ollama.com/install.sh | sh   (or the Mac app from ollama.com/download)
```

2. Ollama docs: agents and coding tools should get at least 64000 tokens of context:

```bash
ollama serve   # skip if the Ollama app is already running; for agents use: OLLAMA_CONTEXT_LENGTH=64000 ollama serve
```

3. Run:

```bash
ollama pull gemma4   # pick your size from the fit chart, e.g. qwen3.5:4b on 8GB, qwen3-coder:30b on 64GB+
```

4. Option B (GUI): LM Studio. Model search names are untested on Mac Studio:

```bash
brew install --cask lm-studio && lms get gemma-4   # lms get searches; pick a Gemma 4 or Qwen 3.5 build (--mlx for MLX on Apple Silicon)
```

5. Run:

```bash
lms server start && lms load <model-id-from-lms-ls>
```

### Commands: Linux

1. Run:

```bash
curl -fsSL https://ollama.com/install.sh | sh && ollama pull gemma4
```

### Commands: Windows

1. Run:

```bash
irm https://ollama.com/install.ps1 | iex; ollama pull gemma4
```

### Verify

1. Expect: a short greeting

```bash
ollama run gemma4 "Say hi in five words."
```

2. Expect: JSON list including gemma4 (Ollama's OpenAI-compatible endpoint)

```bash
curl -s http://localhost:11434/v1/models
```

3. Expect: model listed; PROCESSOR should read 100% GPU (per Ollama context-length docs)

```bash
ollama ps
```

4. Expect: LM Studio path: JSON model list (port 1234 is LM Studio's usual default)

```bash
curl -s http://localhost:1234/v1/models
```

### Rollback

1. Run:

```bash
ollama rm gemma4 && brew uninstall ollama
```

2. Run:

```bash
lms server stop && brew uninstall --cask lm-studio   # models live under ~/.lmstudio
```

Related listings: https://insidethemachine.ai/setups/ollama.md, https://insidethemachine.ai/setups/lm-studio.md

## Step 3: Point an agent at it

**Goal:** A harness (OpenCode, Goose, Aider or Hermes Agent) talks to the local endpoint http://localhost:11434/v1 (Ollama) or http://localhost:1234/v1 (LM Studio).

### Harness options

- **OpenCode**: Terminal coding agent; Ollama can configure it in one command. Docs: https://docs.ollama.com/integrations/opencode
- **Goose**: General agent with MCP extensions; desktop app or CLI. Docs: https://docs.ollama.com/integrations/goose
- **Aider**: Git-aware pair programmer for existing repos. Docs: https://aider.chat/docs/llms/ollama.html
- **Hermes Agent**: Nous Research agent with skills and cross-session memory. Docs: https://docs.ollama.com/integrations/hermes

### Commands: macOS (Apple Silicon)

1. OpenCode (recommended):

```bash
curl -fsSL https://opencode.ai/install | bash   # or: brew install anomalyco/tap/opencode
```

2. starts OpenCode with an inline config for the Ollama model you pick; does not overwrite ~/.config/opencode/opencode.json:

```bash
ollama launch opencode
```

3. manual alternative, format from opencode.ai/docs/providers; model ids must match GET /v1/models. Back up any existing file first:

```bash
mkdir -p ~/.config/opencode && cat > ~/.config/opencode/opencode.json <<'EOF'
{
  "$schema": "https://opencode.ai/config.json",
  "provider": {
    "ollama": {
      "npm": "@ai-sdk/openai-compatible",
      "name": "Ollama (local)",
      "options": { "baseURL": "http://localhost:11434/v1" },
      "models": { "gemma4": { "name": "Gemma 4" } }
    }
  }
}
EOF
```

4. Goose: choose Configure Providers → Ollama → enter your model name:

```bash
curl -fsSL https://github.com/aaif-goose/goose/releases/download/stable/download_cli.sh | bash && goose configure
```

5. Aider: see /setups/aider:

```bash
python -m pip install aider-install && aider-install && aider --model ollama_chat/gemma4
```

6. Hermes Agent: Ollama points it at http://127.0.0.1:11434/v1:

```bash
ollama launch hermes   # or: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
```

### Verify

1. Expect: prints 'ready' using the local model (untested on Mac Studio)

```bash
opencode run "Reply with the word ready"
```

2. Expect: Goose opens a session; type: say ready

```bash
goose session -n itm-check
```

### Rollback

1. Run:

```bash
rm ~/.config/opencode/opencode.json   # restore your backup if you had one
```

2. Run:

```bash
brew uninstall opencode   # if installed via brew
```

3. Run:

```bash
# Goose / Hermes: follow uninstall steps in their docs (install scripts place a binary in your PATH)
```

Related listings: https://insidethemachine.ai/setups/aider.md, https://insidethemachine.ai/setups/ollama.md

## Step 4: Your goal becomes workflows

**Goal:** A workspace with AGENTS.md, memory/MEMORY.md, one folder per workflow and MCP tools configured.

### Default workflows

- research: fetch MCP server (uvx mcp-server-fetch)
- writing: AGENTS.md style rules
- files: filesystem MCP server
- memory: memory/MEMORY.md, upgrade to Mem0
- documents: Docling

### Commands: macOS (Apple Silicon)

1. Run:

```bash
mkdir -p ~/itm-agent/{workflows/{research,writing,files,documents},memory,tools} && cd ~/itm-agent && git init -q
```

2. Run:

```bash
cat > AGENTS.md <<'EOF'
# AGENTS.md
## Goal
<the user's goal in one sentence>
## Workflows
- research/: gather sources, save notes as markdown with URLs
- writing/: drafts; never publish or send without the user's OK
- files/: only touch files inside ~/itm-agent
- documents/: convert PDFs/Office files with `docling <file>`
## Memory
- Read memory/MEMORY.md at the start of every session; append decisions and facts at the end.
## Upgrades
- Before installing anything, read the listing at https://insidethemachine.ai/setups/<slug>.md and run its verify step.
EOF
```

3. Run:

```bash
printf '# MEMORY\n\n- %s: workspace created\n' "$(date +%F)" > memory/MEMORY.md
```

4. project-level OpenCode config (format from opencode.ai/docs/mcp-servers); replace /Users/YOU. Requires uv (brew install uv) and Node:

```bash
cat > opencode.json <<'EOF'
{
  "$schema": "https://opencode.ai/config.json",
  "mcp": {
    "fetch": { "type": "local", "command": ["uvx", "mcp-server-fetch"], "enabled": true },
    "filesystem": { "type": "local", "command": ["npx", "-y", "@modelcontextprotocol/server-filesystem", "/Users/YOU/itm-agent"], "enabled": true }
  }
}
EOF
```

### Verify

1. Expect: AGENTS.md, memory/MEMORY.md, workflows/* folders, opencode.json

```bash
find ~/itm-agent -maxdepth 2 -not -path '*/.git*'
```

2. Expect: the agent lists the workflows (untested on Mac Studio)

```bash
cd ~/itm-agent && opencode run "Read AGENTS.md and list the workflows"
```

### Rollback

1. Run:

```bash
rm -rf ~/itm-agent   # deletes the workspace, irreversible
```

Related listings: https://insidethemachine.ai/setups/agents-md.md, https://insidethemachine.ai/setups/mcp-servers.md, https://insidethemachine.ai/setups/mem0.md, https://insidethemachine.ai/setups/docling.md

## Step 5: It builds and upgrades itself

**Goal:** The agent pulls /llms.txt, installs missing pieces using each listing's verify/rollback, routes via LiteLLM (fast vs smart), and logs a self-check.

### Commands: macOS (Apple Silicon)

1. the agent's map of what exists:

```bash
curl -s https://insidethemachine.ai/llms.txt && curl -s https://insidethemachine.ai/api/index.json
```

2. any listing as markdown:

```bash
curl -s -H 'Accept: text/markdown' https://insidethemachine.ai/setups/litellm
```

3. LiteLLM docs show pip install 'litellm[proxy]'; uv tool install is the isolated equivalent (untested on Mac Studio):

```bash
brew install uv && uv tool install 'litellm[proxy]'
```

4. 128GB example; on smaller machines map smart to the biggest model your chart allows, or to a cloud model you have a key for:

```bash
cat > ~/itm-agent/tools/litellm.yaml <<'EOF'
model_list:
  - model_name: "fast"
    litellm_params:
      model: "ollama_chat/qwen3-coder:30b"
  - model_name: "smart"
    litellm_params:
      model: "ollama_chat/qwen3.5:122b"
EOF
```

5. Run:

```bash
litellm --config ~/itm-agent/tools/litellm.yaml   # OpenAI-compatible proxy on http://0.0.0.0:4000
```

### Verify

1. Expect: OpenAI-format JSON reply

```bash
curl -s http://0.0.0.0:4000/chat/completions -H 'Content-Type: application/json' -d '{"model":"fast","messages":[{"role":"user","content":"ready?"}]}'
```

2. Expect: the agent logged what it installed and each verify result

```bash
tail -5 ~/itm-agent/memory/MEMORY.md
```

### Rollback

1. Run:

```bash
uv tool uninstall litellm && rm ~/itm-agent/tools/litellm.yaml
```

2. Run:

```bash
# for anything else the agent installed: run the rollback section of that listing's .md
```

### Self-check

Official Gap Scores are pending test on Mac Studio. Until then the agent's self-check is: did every listing's verify step pass, plus tokens/sec from `ollama run <model> --verbose` (untested on Mac Studio).

Related listings: https://insidethemachine.ai/setups/litellm.md, https://insidethemachine.ai/setups/mem0.md, https://insidethemachine.ai/setups/open-webui.md

## Hermes-style gaps: identity, instructions, and lean context

Frontier apps feel smart partly because of what surrounds the model: an identity, project rules, skills, memory and careful context management. These four parts give a local agent the same scaffolding without a million-token window.

### AGENTS.md and SOUL.md

#### AGENTS.md = how to work here

A plain markdown file of project rules your agent reads every session. Most harnesses (OpenCode, Goose, Hermes, OpenClaw) load it automatically.

- AGENTS.md: open, harness-agnostic project-instructions file (see /setups/agents-md).
- Hermes: loads AGENTS.md from the CWD at startup and subdirectories progressively; .hermes.md / HERMES.md take highest priority (docs: user-guide/features/context-files).
- OpenCode: reads AGENTS.md in the project and ~/.config/opencode/AGENTS.md globally (opencode.ai/docs/rules).
- Goose: CONTEXT_FILE_NAMES default is ["AGENTS.md", ".goosehints"] (goose-docs.ai using-goosehints).
- OpenClaw: workspace files include AGENTS.md, SOUL.md, IDENTITY, USER (docs.openclaw.ai templates); default workspace ~/.openclaw/workspace.

Template (~/itm-agent/AGENTS.md):

```markdown
# AGENTS.md
<!-- Always loaded. Keep under ~150 lines: rules + pointers, not knowledge. -->
## Project
One sentence: what this workspace is for.
## Rules
- Work only inside this folder. Ask before deleting, spending, or sending anything.
- Prefer small diffs. Run the verify step after every install.
## Map (load on demand, do not paste here)
- workflows/<name>/AGENTS.md : rules for that workflow only
- memory/MEMORY.md : read the last 30 lines at session start; append decisions
- skills/ : one folder per repeatable procedure (SKILL.md each)
## Output hygiene
- Pipe long command output through `tail -n 50` or `head -c 4000`.
- Summarize files; never print whole logs or datasets.
```

#### SOUL.md = who the agent is

A persona file: voice, values and limits. Hermes puts it first in the system prompt; OpenClaw injects it into normal sessions.

- Hermes: SOUL.md is the primary identity, slot #1 in the system prompt, loaded only from $HERMES_HOME/SOUL.md (default ~/.hermes/SOUL.md). Hermes seeds a starter file and never overwrites yours (docs: features/personality).
- OpenClaw: 'SOUL.md is where your agent's voice lives. OpenClaw injects it into normal sessions' (docs.openclaw.ai/concepts/soul). Lives in the agent workspace.
- OpenCode / Goose / Aider have no SOUL.md concept: put persona lines at the top of AGENTS.md or list SOUL.md as an extra instruction file (below).

Template (~/.hermes/SOUL.md):

```markdown
# SOUL.md
<!-- Identity, voice, boundaries. No project facts: those go in AGENTS.md. -->
You are Kit, a calm, direct assistant running on a local model.
## Voice
- Short sentences. Lead with the answer. No filler, no hype.
- Say "I don't know" and check, rather than guess.
## Values
- The user's machine and data stay local unless they say otherwise.
- Ask before irreversible actions.
## Never
- Invent numbers, benchmarks or citations.
```

### Point each harness at the files

#### Hermes

Edit ~/.hermes/SOUL.md for identity and start Hermes inside your workspace so it finds AGENTS.md. Skills live in ~/.hermes/skills.

```bash
$EDITOR ~/.hermes/SOUL.md
cd ~/itm-agent && hermes
```

- Note: Directory layout per docs: ~/.hermes/{config.yaml,SOUL.md,memories/,skills/}.

#### OpenCode

It reads AGENTS.md on its own. Add extra files like SOUL.md with one line of config.

```bash
cat > ~/itm-agent/opencode.json <<'EOF'
{
  "$schema": "https://opencode.ai/config.json",
  "instructions": ["SOUL.md", "workflows/*/AGENTS.md"]
}
EOF
```

- Note: 'instructions' accepts paths, globs and URLs; combined with AGENTS.md (opencode.ai/docs/rules). Merge with the mcp block from step 4 if you created one.

#### Goose

Goose reads AGENTS.md and .goosehints by default. Tell it about other files with one environment variable.

```bash
export CONTEXT_FILE_NAMES='["AGENTS.md", "SOUL.md", ".goosehints"]'
cd ~/itm-agent && goose session -n itm
```

- Note: Global hints: ~/.config/goose/.goosehints; local hints win over global.

#### Aider

Aider loads rule files as read-only context. Put them in its config so every session gets them.

```bash
printf 'read: [AGENTS.md, SOUL.md]\n' > ~/itm-agent/.aider.conf.yml
aider --read AGENTS.md   # one-off equivalent
```

- Note: Docs: aider.chat/docs/usage/conventions.html (read files are marked read-only and cached if prompt caching is enabled).

### Avoid context bloat

#### Keep always-loaded files lean

AGENTS.md and SOUL.md ride along on every turn, so every line costs you. Keep rules and pointers there, and nothing else.

- AGENTS.md: aim for under ~150 lines (house rule of thumb, not a harness limit)
- SOUL.md: aim for under ~40 lines

#### Skills load only when needed

Write repeatable procedures as skills: the agent sees a one-line description and opens the full text only when it needs it. Hermes calls this progressive disclosure.

- Hermes skills: on-demand documents following a progressive disclosure pattern to minimize token usage; agentskills.io compatible; stored in ~/.hermes/skills/.

Layout: skills/<name>/SKILL.md with front-matter name + description; body loaded on demand

#### Retrieve, don't stuff

Keep notes and documents on disk and let the agent search them. Pull in a few relevant chunks instead of whole files.

- memory/MEMORY.md tail + grep
- Mem0 (/setups/mem0)
- Docling to convert PDFs before retrieval (/setups/docling)
- Open WebUI knowledge bases (/setups/open-webui)

#### Compact long sessions

When a chat gets long, have the harness summarize the middle and keep the recent tail. Turn it on before the window fills.

opencode.json:

```
{
  "compaction": {
    "auto": true,
    "prune": true,
    "reserved": 10000
  }
}
```

~/.hermes/config.yaml:

```
compression:
  enabled: true
  threshold: 0.50   # compress at 50% of context limit
  target_ratio: 0.20
```

hermes (manual):

```
/compress
```

aider (manual):

```
/clear or /drop unused files
```

- Note: OpenCode prune=true removes old tool outputs (default false). Hermes: the summary model needs a context window at least as large as the main model's.

#### Trim tool output

Most bloat is logs and command output, not your words. Cap it at the source.

```bash
<cmd> 2>&1 | tail -n 50
<cmd> | head -c 4000
rg -n --max-count 20 'pattern' .
```

#### One folder per workflow

Give each workflow its own folder and its own small AGENTS.md. The agent only loads the rules for the job in front of it.

Layout: ~/itm-agent/workflows/{research,writing,files,documents}/AGENTS.md; start the harness inside the workflow folder (Hermes and OpenCode both pick up subdirectory AGENTS.md)

### Keep the local model efficient

#### Pick the right quant

A 4-bit model is about a quarter the size of 16-bit and usually close in quality. Use the fit chart and leave room for context.

- ollama default tags: 4-bit-class quants (sizes in step 1 chart)
- rule: prefer a bigger model at 4-bit over a smaller one at 8-bit when it fits (rule of thumb, untested on Mac Studio)

#### Set context length on purpose

Bigger windows eat memory fast. Give agents enough (64k) and no more.

ollama:

```bash
OLLAMA_CONTEXT_LENGTH=64000 ollama serve   # server-wide
/set parameter num_ctx 64000   # inside ollama run
API: "options": {"num_ctx": 64000}
ollama ps   # CONTEXT column shows what was allocated
```

llama.cpp:

```bash
llama-server -m model.gguf -c 65536   # -c, --ctx-size; 0 = from model
```

mlx-lm:

```bash
mlx_lm.generate --max-kv-size 65536 ...   # rotating fixed-size KV cache
```

- Note: Ollama: RAM scales with OLLAMA_NUM_PARALLEL × OLLAMA_CONTEXT_LENGTH; keep OLLAMA_NUM_PARALLEL=1 for a single agent.

#### Shrink the KV cache

The cache that holds your context can be quantized too. 8-bit halves its memory with very little quality loss.

ollama:

```bash
OLLAMA_FLASH_ATTENTION=1 OLLAMA_KV_CACHE_TYPE=q8_0 ollama serve
# values: f16 (default), q8_0 (~1/2 memory, recommended), q4_0 (~1/4, more loss at long context); global for all models
```

llama.cpp:

```bash
llama-server -m model.gguf -c 65536 -fa on -ctk q8_0 -ctv q8_0
```

- Note: llama.cpp allowed K/V types: f32, f16, bf16, q8_0, q4_0, q4_1, iq4_nl, q5_0, q5_1 (default f16).

#### Reuse the prompt

Your AGENTS.md and SOUL.md are the same every turn, so cache them instead of re-reading. Keep them stable so the cache keeps hitting.

ollama:

```bash
OLLAMA_KEEP_ALIVE=30m ollama serve   # keep the model loaded between turns
```

llama.cpp:

```bash
llama-server ... --cache-reuse 256   # reuse cached chunks via KV shifting; needs prompt caching
```

mlx-lm:

```bash
cat AGENTS.md | mlx_lm.cache_prompt --model <model> --prompt - --prompt-cache-file agents.safetensors
mlx_lm.generate --prompt-cache-file agents.safetensors --prompt "..."
```

- Note: Editing the top of an always-loaded file invalidates the cached prefix: put changing content at the end.

#### Small model first, big model when needed

Send routine turns to a fast model and hard ones to your biggest. LiteLLM gives both one address.

See: Step 5 litellm.yaml (fast / smart aliases)

- Note: Hermes can also run compression on a separate model via auxiliary.compression (model, base_url).

## Sources

- https://aider.chat/docs/llms/ollama.html
- https://aider.chat/docs/usage/conventions.html
- https://docs.litellm.ai/docs/providers/ollama
- https://docs.ollama.com/context-length
- https://docs.ollama.com/faq
- https://docs.ollama.com/integrations/goose
- https://docs.ollama.com/integrations/hermes
- https://docs.ollama.com/integrations/opencode
- https://docs.openclaw.ai/concepts/agent-workspace
- https://docs.openclaw.ai/concepts/soul
- https://github.com/aaif-goose/goose
- https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md
- https://github.com/ml-explore/mlx-lm
- https://github.com/modelcontextprotocol/servers
- https://goose-docs.ai/docs/getting-started/providers
- https://goose-docs.ai/docs/guides/context-engineering/using-goosehints
- https://hermes-agent.nousresearch.com/docs/user-guide/configuration
- https://hermes-agent.nousresearch.com/docs/user-guide/features/context-files
- https://hermes-agent.nousresearch.com/docs/user-guide/features/personality
- https://hermes-agent.nousresearch.com/docs/user-guide/features/skills
- https://lmstudio.ai/docs/cli
- https://ollama.com/library/gemma4/tags
- https://ollama.com/library/qwen3-coder/tags
- https://ollama.com/library/qwen3.5/tags
- https://opencode.ai/docs/cli/
- https://opencode.ai/docs/config/
- https://opencode.ai/docs/mcp-servers/
- https://opencode.ai/docs/providers/
- https://opencode.ai/docs/rules/