Inside The Machine

Advanced guide · macOS first

Build an agent that improves itself.

Five steps. Your agent does most of them.

New to local AI? Start with a recipe first. This guide builds on the agent recipe.

  1. 1Check your machine
  2. 2Pick your tool
  3. 3Point an agent at it
  4. 4Your goal becomes workflows
  5. 5It builds and upgrades itself
1Step 1 of 5

Check your machine

Your memory decides which models you can run. Find your number, then read across the chart.

MemoryQwen 3.5 4B3.3 GBGemma 46.6 GBGemma 4 26B16 GBQwen3 Coder 30B19 GBQwen 3.5 122B81 GB
8 GB
16 GB
32 GB
64 GB
128 GBtest rig
fitstighttoo bigEstimate: model ≤ half your memory. Not measured.
2Step 2 of 5

Pick your tool

Install Ollama (terminal) or LM Studio (app), then download one model. Say hi; if it answers, you're running local AI.

OllamaTerminal · best for agents
or
LM StudioApp · point and click
Say hi.
Hi! I'm running on your machine.
3Step 3 of 5

Point an agent at it

Install an agent harness and connect it to your local model. Then just tell it what you want, in plain English.

  • OpenCode
  • Goose
  • Aider
  • Hermes
localhost:11434Your model
“Set me up to research and write about local AI.”
4Step 4 of 5

Your goal becomes workflows

The agent splits your goal into workflows like research, writing, files and memory. It sets up folders, an AGENTS.md, a memory file and the tools each one needs.

Your goal
  • research
  • writing
  • files
  • memory
  • documents
AGENTS.mdMEMORY.md5 foldersMCP tools
5Step 5 of 5

It builds and upgrades itself

Your agent reads this site's agent pages, installs what's missing and sends hard jobs to a bigger model. Then it re-checks how close it is to the frontier.

Readllms.txt
Installwhat's missing
Routefast · smart
Re-checkverify + log
RepeatGap Score: pending
Hermes-style gaps

Give it an identity, rules, and a lean context.

AGENTS.mdHow to work hererules + pointers · every harness
SOUL.mdWho the agent isvoice + limits · Hermes, OpenClaw
Your agentloaded every turn, so keep both short

Point each harness at the files

  • Hermes

    Edit ~/.hermes/SOUL.md for identity and start Hermes inside your workspace so it finds AGENTS.md. Skills live in ~/.hermes/skills.

  • OpenCode

    It reads AGENTS.md on its own. Add extra files like SOUL.md with one line of config.

  • Goose

    Goose reads AGENTS.md and .goosehints by default. Tell it about other files with one environment variable.

  • Aider

    Aider loads rule files as read-only context. Put them in its config so every session gets them.

Avoid context bloat

  • Keep always-loaded files lean

    AGENTS.md and SOUL.md ride along on every turn, so every line costs you. Keep rules and pointers there, and nothing else.

  • Skills load only when needed

    Write repeatable procedures as skills: the agent sees a one-line description and opens the full text only when it needs it. Hermes calls this progressive disclosure.

  • Retrieve, don't stuff

    Keep notes and documents on disk and let the agent search them. Pull in a few relevant chunks instead of whole files.

  • Compact long sessions

    When a chat gets long, have the harness summarize the middle and keep the recent tail. Turn it on before the window fills.

  • Trim tool output

    Most bloat is logs and command output, not your words. Cap it at the source.

  • One folder per workflow

    Give each workflow its own folder and its own small AGENTS.md. The agent only loads the rules for the job in front of it.

Keep the local model efficient

  • Pick the right quant

    A 4-bit model is about a quarter the size of 16-bit and usually close in quality. Use the fit chart and leave room for context.

  • Set context length on purpose

    Bigger windows eat memory fast. Give agents enough (64k) and no more.

  • Shrink the KV cache

    The cache that holds your context can be quantized too. 8-bit halves its memory with very little quality loss.

  • Reuse the prompt

    Your AGENTS.md and SOUL.md are the same every turn, so cache them instead of re-reading. Keep them stable so the cache keeps hitting.

  • Small model first, big model when needed

    Send routine turns to a fast model and hard ones to your biggest. LiteLLM gives both one address.

Harness spotlight

Hermes Agent

A personal agent that learns skills and remembers you.

Your local model gets a learning loop, not just a chat box.

  • Memory across sessions
  • Writes its own skills
  • Chat from Telegram, Slack…
$ ollama launch hermesFull listing
Does the task
Saves a skill
Remembers you
Reuses it next time
The shortcut

Paste one prompt. Your agent runs steps 4 and 5.

Works once step 3 is done.

You are my local AI setup agent. Work only inside ~/itm-agent. Ask before deleting anything, spending money, or sending anything outside this machine.

Context: a local model server is already running (Ollama at http://localhost:11434/v1 or LM Studio at http://localhost:1234/v1).

1. Read https://insidethemachine.ai/start.md and https://insidethemachine.ai/llms.txt. Use them as your runbook and catalog.
Commands checked against official docs 2026-10-15. Not yet run end-to-end on the Mac Studio.
AGENT VIEW · text/markdown · or send Accept: text/markdown
---
title: "Zero to working local AI"
slug: start
kind: runbook
verified_on: 2026-10-15
status: "Commands verified against official docs on 2026-10-15; not yet run end-to-end on the Mac Studio test rig unless marked mac_studio_tested: true."
platform_priority: ["macos-apple-silicon", "linux", "windows"]
steps: 5
json: https://insidethemachine.ai/start.json
catalog: https://insidethemachine.ai/api/index.json
---

# Zero to working local AI

> Executable runbook. Run steps in order; each has goal, commands per platform, verify (with expected output) and rollback. Commands verified against official docs on 2026-10-15; not yet run end-to-end on the Mac Studio test rig unless marked mac_studio_tested: true.

## Bootstrap prompt

Paste this into your agent once step 3 works:

```text
You are my local AI setup agent. Work only inside ~/itm-agent. Ask before deleting anything, spending money, or sending anything outside this machine.

Context: a local model server is already running (Ollama at http://localhost:11434/v1 or LM Studio at http://localhost:1234/v1).

1. Read https://insidethemachine.ai/start.md and https://insidethemachine.ai/llms.txt. Use them as your runbook and catalog.
2. My goal: <describe what you want, in plain English>.
3. Split that goal into workflows (for example research, writing, files, documents, memory). Create ~/itm-agent with AGENTS.md (goal, workflows, rules), memory/MEMORY.md, and one folder per workflow, following step 4 of start.md.
4. For each workflow, pick the tools it needs from the catalog. For each tool, fetch https://insidethemachine.ai/setups/<slug>.md, run its install and verify steps, and write the result to memory/MEMORY.md. If a verify step fails, run that listing's rollback and tell me.
5. Set up LiteLLM routing (step 5): "fast" = my everyday local model, "smart" = the biggest model my RAM allows. Use smart only for hard tasks.
6. Finish with a short report: what you installed, what passed, what failed, and what you'd upgrade next. Note that official Gap Scores are still pending test on Mac Studio.
```

## Step 1: Check your machine

**Goal:** Know total RAM (Apple Silicon: unified memory) or GPU VRAM, and pick a model size from the fit chart.

### Fit chart

Rule of thumb (estimate, not measured): comfortable if the model file is at most 50% of RAM/VRAM, tight up to 75%, otherwise no. Context window and the OS need the rest. Sizes: https://ollama.com/library (tags pages, download sizes as listed 2026-10-15)

| RAM | qwen3.5:4b (3.3 GB) | gemma4 (6.6 GB) | gemma4:26b (16 GB) | qwen3-coder:30b (19 GB) | qwen3.5:122b (81 GB) | pick |
|---|---|---|---|---|---|---|
| 8 GB | comfortable | no | no | no | no | qwen3.5:4b |
| 16 GB | comfortable | comfortable | no | no | no | gemma4 |
| 32 GB | comfortable | comfortable | comfortable | tight | no | gemma4:26b |
| 64 GB | comfortable | comfortable | comfortable | comfortable | no | qwen3-coder:30b |
| 128 GB | comfortable | comfortable | comfortable | comfortable | tight | qwen3-coder:30b (fast) + qwen3.5:122b (smart) |

### Commands: macOS (Apple Silicon)

1. Run:

```bash
sysctl -n hw.memsize | awk '{printf "%.0f GB unified memory\n", $1/1073741824}'
```

2. Run:

```bash
system_profiler SPHardwareDataType | grep -E 'Chip|Memory'
```

### Commands: Linux

1. Run:

```bash
free -g | awk '/Mem:/{print $2" GB RAM"}'
```

2. Run:

```bash
nvidia-smi --query-gpu=name,memory.total --format=csv   # NVIDIA GPUs only
```

### Commands: Windows

1. Run:

```bash
(Get-CimInstance Win32_ComputerSystem).TotalPhysicalMemory/1GB   # PowerShell
```

### Verify

1. Expect: prints a byte count; 128GB Mac Studio ≈ 137438953472

```bash
sysctl -n hw.memsize
```

### Rollback

1. Run:

```bash
# read-only step; nothing to undo
```

Guides: Machines: what to buy or what yours can run -> https://insidethemachine.ai/machines.md (JSON: https://insidethemachine.ai/machines.json); Models vs memory: what fits your RAM -> https://insidethemachine.ai/models.md (JSON: https://insidethemachine.ai/models.json)

Related listings: https://insidethemachine.ai/setups/mlx-lm.md, https://insidethemachine.ai/setups/llama-cpp.md

## Step 2: Pick your tool

**Goal:** A local model server is running and answers a prompt.

### Commands: macOS (Apple Silicon)

1. Option A (recommended for agents): Ollama:

```bash
brew install ollama   # or: curl -fsSL https://ollama.com/install.sh | sh   (or the Mac app from ollama.com/download)
```

2. Ollama docs: agents and coding tools should get at least 64000 tokens of context:

```bash
ollama serve   # skip if the Ollama app is already running; for agents use: OLLAMA_CONTEXT_LENGTH=64000 ollama serve
```

3. Run:

```bash
ollama pull gemma4   # pick your size from the fit chart, e.g. qwen3.5:4b on 8GB, qwen3-coder:30b on 64GB+
```

4. Option B (GUI): LM Studio. Model search names are untested on Mac Studio:

```bash
brew install --cask lm-studio && lms get gemma-4   # lms get searches; pick a Gemma 4 or Qwen 3.5 build (--mlx for MLX on Apple Silicon)
```

5. Run:

```bash
lms server start && lms load <model-id-from-lms-ls>
```

### Commands: Linux

1. Run:

```bash
curl -fsSL https://ollama.com/install.sh | sh && ollama pull gemma4
```

### Commands: Windows

1. Run:

```bash
irm https://ollama.com/install.ps1 | iex; ollama pull gemma4
```

### Verify

1. Expect: a short greeting

```bash
ollama run gemma4 "Say hi in five words."
```

2. Expect: JSON list including gemma4 (Ollama's OpenAI-compatible endpoint)

```bash
curl -s http://localhost:11434/v1/models
```

3. Expect: model listed; PROCESSOR should read 100% GPU (per Ollama context-length docs)

```bash
ollama ps
```

4. Expect: LM Studio path: JSON model list (port 1234 is LM Studio's usual default)

```bash
curl -s http://localhost:1234/v1/models
```

### Rollback

1. Run:

```bash
ollama rm gemma4 && brew uninstall ollama
```

2. Run:

```bash
lms server stop && brew uninstall --cask lm-studio   # models live under ~/.lmstudio
```

Related listings: https://insidethemachine.ai/setups/ollama.md, https://insidethemachine.ai/setups/lm-studio.md

## Step 3: Point an agent at it

**Goal:** A harness (OpenCode, Goose, Aider or Hermes Agent) talks to the local endpoint http://localhost:11434/v1 (Ollama) or http://localhost:1234/v1 (LM Studio).

### Harness options

- **OpenCode**: Terminal coding agent; Ollama can configure it in one command. Docs: https://docs.ollama.com/integrations/opencode
- **Goose**: General agent with MCP extensions; desktop app or CLI. Docs: https://docs.ollama.com/integrations/goose
- **Aider**: Git-aware pair programmer for existing repos. Docs: https://aider.chat/docs/llms/ollama.html
- **Hermes Agent**: Nous Research agent with skills and cross-session memory. Docs: https://docs.ollama.com/integrations/hermes

### Commands: macOS (Apple Silicon)

1. OpenCode (recommended):

```bash
curl -fsSL https://opencode.ai/install | bash   # or: brew install anomalyco/tap/opencode
```

2. starts OpenCode with an inline config for the Ollama model you pick; does not overwrite ~/.config/opencode/opencode.json:

```bash
ollama launch opencode
```

3. manual alternative, format from opencode.ai/docs/providers; model ids must match GET /v1/models. Back up any existing file first:

```bash
mkdir -p ~/.config/opencode && cat > ~/.config/opencode/opencode.json <<'EOF'
{
  "$schema": "https://opencode.ai/config.json",
  "provider": {
    "ollama": {
      "npm": "@ai-sdk/openai-compatible",
      "name": "Ollama (local)",
      "options": { "baseURL": "http://localhost:11434/v1" },
      "models": { "gemma4": { "name": "Gemma 4" } }
    }
  }
}
EOF
```

4. Goose: choose Configure Providers → Ollama → enter your model name:

```bash
curl -fsSL https://github.com/aaif-goose/goose/releases/download/stable/download_cli.sh | bash && goose configure
```

5. Aider: see /setups/aider:

```bash
python -m pip install aider-install && aider-install && aider --model ollama_chat/gemma4
```

6. Hermes Agent: Ollama points it at http://127.0.0.1:11434/v1:

```bash
ollama launch hermes   # or: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
```

### Verify

1. Expect: prints 'ready' using the local model (untested on Mac Studio)

```bash
opencode run "Reply with the word ready"
```

2. Expect: Goose opens a session; type: say ready

```bash
goose session -n itm-check
```

### Rollback

1. Run:

```bash
rm ~/.config/opencode/opencode.json   # restore your backup if you had one
```

2. Run:

```bash
brew uninstall opencode   # if installed via brew
```

3. Run:

```bash
# Goose / Hermes: follow uninstall steps in their docs (install scripts place a binary in your PATH)
```

Related listings: https://insidethemachine.ai/setups/aider.md, https://insidethemachine.ai/setups/ollama.md

## Step 4: Your goal becomes workflows

**Goal:** A workspace with AGENTS.md, memory/MEMORY.md, one folder per workflow and MCP tools configured.

### Default workflows

- research: fetch MCP server (uvx mcp-server-fetch)
- writing: AGENTS.md style rules
- files: filesystem MCP server
- memory: memory/MEMORY.md, upgrade to Mem0
- documents: Docling

### Commands: macOS (Apple Silicon)

1. Run:

```bash
mkdir -p ~/itm-agent/{workflows/{research,writing,files,documents},memory,tools} && cd ~/itm-agent && git init -q
```

2. Run:

```bash
cat > AGENTS.md <<'EOF'
# AGENTS.md
## Goal
<the user's goal in one sentence>
## Workflows
- research/: gather sources, save notes as markdown with URLs
- writing/: drafts; never publish or send without the user's OK
- files/: only touch files inside ~/itm-agent
- documents/: convert PDFs/Office files with `docling <file>`
## Memory
- Read memory/MEMORY.md at the start of every session; append decisions and facts at the end.
## Upgrades
- Before installing anything, read the listing at https://insidethemachine.ai/setups/<slug>.md and run its verify step.
EOF
```

3. Run:

```bash
printf '# MEMORY\n\n- %s: workspace created\n' "$(date +%F)" > memory/MEMORY.md
```

4. project-level OpenCode config (format from opencode.ai/docs/mcp-servers); replace /Users/YOU. Requires uv (brew install uv) and Node:

```bash
cat > opencode.json <<'EOF'
{
  "$schema": "https://opencode.ai/config.json",
  "mcp": {
    "fetch": { "type": "local", "command": ["uvx", "mcp-server-fetch"], "enabled": true },
    "filesystem": { "type": "local", "command": ["npx", "-y", "@modelcontextprotocol/server-filesystem", "/Users/YOU/itm-agent"], "enabled": true }
  }
}
EOF
```

### Verify

1. Expect: AGENTS.md, memory/MEMORY.md, workflows/* folders, opencode.json

```bash
find ~/itm-agent -maxdepth 2 -not -path '*/.git*'
```

2. Expect: the agent lists the workflows (untested on Mac Studio)

```bash
cd ~/itm-agent && opencode run "Read AGENTS.md and list the workflows"
```

### Rollback

1. Run:

```bash
rm -rf ~/itm-agent   # deletes the workspace, irreversible
```

Related listings: https://insidethemachine.ai/setups/agents-md.md, https://insidethemachine.ai/setups/mcp-servers.md, https://insidethemachine.ai/setups/mem0.md, https://insidethemachine.ai/setups/docling.md

## Step 5: It builds and upgrades itself

**Goal:** The agent pulls /llms.txt, installs missing pieces using each listing's verify/rollback, routes via LiteLLM (fast vs smart), and logs a self-check.

### Commands: macOS (Apple Silicon)

1. the agent's map of what exists:

```bash
curl -s https://insidethemachine.ai/llms.txt && curl -s https://insidethemachine.ai/api/index.json
```

2. any listing as markdown:

```bash
curl -s -H 'Accept: text/markdown' https://insidethemachine.ai/setups/litellm
```

3. LiteLLM docs show pip install 'litellm[proxy]'; uv tool install is the isolated equivalent (untested on Mac Studio):

```bash
brew install uv && uv tool install 'litellm[proxy]'
```

4. 128GB example; on smaller machines map smart to the biggest model your chart allows, or to a cloud model you have a key for:

```bash
cat > ~/itm-agent/tools/litellm.yaml <<'EOF'
model_list:
  - model_name: "fast"
    litellm_params:
      model: "ollama_chat/qwen3-coder:30b"
  - model_name: "smart"
    litellm_params:
      model: "ollama_chat/qwen3.5:122b"
EOF
```

5. Run:

```bash
litellm --config ~/itm-agent/tools/litellm.yaml   # OpenAI-compatible proxy on http://0.0.0.0:4000
```

### Verify

1. Expect: OpenAI-format JSON reply

```bash
curl -s http://0.0.0.0:4000/chat/completions -H 'Content-Type: application/json' -d '{"model":"fast","messages":[{"role":"user","content":"ready?"}]}'
```

2. Expect: the agent logged what it installed and each verify result

```bash
tail -5 ~/itm-agent/memory/MEMORY.md
```

### Rollback

1. Run:

```bash
uv tool uninstall litellm && rm ~/itm-agent/tools/litellm.yaml
```

2. Run:

```bash
# for anything else the agent installed: run the rollback section of that listing's .md
```

### Self-check

Official Gap Scores are pending test on Mac Studio. Until then the agent's self-check is: did every listing's verify step pass, plus tokens/sec from `ollama run <model> --verbose` (untested on Mac Studio).

Related listings: https://insidethemachine.ai/setups/litellm.md, https://insidethemachine.ai/setups/mem0.md, https://insidethemachine.ai/setups/open-webui.md

## Hermes-style gaps: identity, instructions, and lean context

Frontier apps feel smart partly because of what surrounds the model: an identity, project rules, skills, memory and careful context management. These four parts give a local agent the same scaffolding without a million-token window.

### AGENTS.md and SOUL.md

#### AGENTS.md = how to work here

A plain markdown file of project rules your agent reads every session. Most harnesses (OpenCode, Goose, Hermes, OpenClaw) load it automatically.

- AGENTS.md: open, harness-agnostic project-instructions file (see /setups/agents-md).
- Hermes: loads AGENTS.md from the CWD at startup and subdirectories progressively; .hermes.md / HERMES.md take highest priority (docs: user-guide/features/context-files).
- OpenCode: reads AGENTS.md in the project and ~/.config/opencode/AGENTS.md globally (opencode.ai/docs/rules).
- Goose: CONTEXT_FILE_NAMES default is ["AGENTS.md", ".goosehints"] (goose-docs.ai using-goosehints).
- OpenClaw: workspace files include AGENTS.md, SOUL.md, IDENTITY, USER (docs.openclaw.ai templates); default workspace ~/.openclaw/workspace.

Template (~/itm-agent/AGENTS.md):

```markdown
# AGENTS.md
<!-- Always loaded. Keep under ~150 lines: rules + pointers, not knowledge. -->
## Project
One sentence: what this workspace is for.
## Rules
- Work only inside this folder. Ask before deleting, spending, or sending anything.
- Prefer small diffs. Run the verify step after every install.
## Map (load on demand, do not paste here)
- workflows/<name>/AGENTS.md : rules for that workflow only
- memory/MEMORY.md : read the last 30 lines at session start; append decisions
- skills/ : one folder per repeatable procedure (SKILL.md each)
## Output hygiene
- Pipe long command output through `tail -n 50` or `head -c 4000`.
- Summarize files; never print whole logs or datasets.
```

#### SOUL.md = who the agent is

A persona file: voice, values and limits. Hermes puts it first in the system prompt; OpenClaw injects it into normal sessions.

- Hermes: SOUL.md is the primary identity, slot #1 in the system prompt, loaded only from $HERMES_HOME/SOUL.md (default ~/.hermes/SOUL.md). Hermes seeds a starter file and never overwrites yours (docs: features/personality).
- OpenClaw: 'SOUL.md is where your agent's voice lives. OpenClaw injects it into normal sessions' (docs.openclaw.ai/concepts/soul). Lives in the agent workspace.
- OpenCode / Goose / Aider have no SOUL.md concept: put persona lines at the top of AGENTS.md or list SOUL.md as an extra instruction file (below).

Template (~/.hermes/SOUL.md):

```markdown
# SOUL.md
<!-- Identity, voice, boundaries. No project facts: those go in AGENTS.md. -->
You are Kit, a calm, direct assistant running on a local model.
## Voice
- Short sentences. Lead with the answer. No filler, no hype.
- Say "I don't know" and check, rather than guess.
## Values
- The user's machine and data stay local unless they say otherwise.
- Ask before irreversible actions.
## Never
- Invent numbers, benchmarks or citations.
```

### Point each harness at the files

#### Hermes

Edit ~/.hermes/SOUL.md for identity and start Hermes inside your workspace so it finds AGENTS.md. Skills live in ~/.hermes/skills.

```bash
$EDITOR ~/.hermes/SOUL.md
cd ~/itm-agent && hermes
```

- Note: Directory layout per docs: ~/.hermes/{config.yaml,SOUL.md,memories/,skills/}.

#### OpenCode

It reads AGENTS.md on its own. Add extra files like SOUL.md with one line of config.

```bash
cat > ~/itm-agent/opencode.json <<'EOF'
{
  "$schema": "https://opencode.ai/config.json",
  "instructions": ["SOUL.md", "workflows/*/AGENTS.md"]
}
EOF
```

- Note: 'instructions' accepts paths, globs and URLs; combined with AGENTS.md (opencode.ai/docs/rules). Merge with the mcp block from step 4 if you created one.

#### Goose

Goose reads AGENTS.md and .goosehints by default. Tell it about other files with one environment variable.

```bash
export CONTEXT_FILE_NAMES='["AGENTS.md", "SOUL.md", ".goosehints"]'
cd ~/itm-agent && goose session -n itm
```

- Note: Global hints: ~/.config/goose/.goosehints; local hints win over global.

#### Aider

Aider loads rule files as read-only context. Put them in its config so every session gets them.

```bash
printf 'read: [AGENTS.md, SOUL.md]\n' > ~/itm-agent/.aider.conf.yml
aider --read AGENTS.md   # one-off equivalent
```

- Note: Docs: aider.chat/docs/usage/conventions.html (read files are marked read-only and cached if prompt caching is enabled).

### Avoid context bloat

#### Keep always-loaded files lean

AGENTS.md and SOUL.md ride along on every turn, so every line costs you. Keep rules and pointers there, and nothing else.

- AGENTS.md: aim for under ~150 lines (house rule of thumb, not a harness limit)
- SOUL.md: aim for under ~40 lines

#### Skills load only when needed

Write repeatable procedures as skills: the agent sees a one-line description and opens the full text only when it needs it. Hermes calls this progressive disclosure.

- Hermes skills: on-demand documents following a progressive disclosure pattern to minimize token usage; agentskills.io compatible; stored in ~/.hermes/skills/.

Layout: skills/<name>/SKILL.md with front-matter name + description; body loaded on demand

#### Retrieve, don't stuff

Keep notes and documents on disk and let the agent search them. Pull in a few relevant chunks instead of whole files.

- memory/MEMORY.md tail + grep
- Mem0 (/setups/mem0)
- Docling to convert PDFs before retrieval (/setups/docling)
- Open WebUI knowledge bases (/setups/open-webui)

#### Compact long sessions

When a chat gets long, have the harness summarize the middle and keep the recent tail. Turn it on before the window fills.

opencode.json:

```
{
  "compaction": {
    "auto": true,
    "prune": true,
    "reserved": 10000
  }
}
```

~/.hermes/config.yaml:

```
compression:
  enabled: true
  threshold: 0.50   # compress at 50% of context limit
  target_ratio: 0.20
```

hermes (manual):

```
/compress
```

aider (manual):

```
/clear or /drop unused files
```

- Note: OpenCode prune=true removes old tool outputs (default false). Hermes: the summary model needs a context window at least as large as the main model's.

#### Trim tool output

Most bloat is logs and command output, not your words. Cap it at the source.

```bash
<cmd> 2>&1 | tail -n 50
<cmd> | head -c 4000
rg -n --max-count 20 'pattern' .
```

#### One folder per workflow

Give each workflow its own folder and its own small AGENTS.md. The agent only loads the rules for the job in front of it.

Layout: ~/itm-agent/workflows/{research,writing,files,documents}/AGENTS.md; start the harness inside the workflow folder (Hermes and OpenCode both pick up subdirectory AGENTS.md)

### Keep the local model efficient

#### Pick the right quant

A 4-bit model is about a quarter the size of 16-bit and usually close in quality. Use the fit chart and leave room for context.

- ollama default tags: 4-bit-class quants (sizes in step 1 chart)
- rule: prefer a bigger model at 4-bit over a smaller one at 8-bit when it fits (rule of thumb, untested on Mac Studio)

#### Set context length on purpose

Bigger windows eat memory fast. Give agents enough (64k) and no more.

ollama:

```bash
OLLAMA_CONTEXT_LENGTH=64000 ollama serve   # server-wide
/set parameter num_ctx 64000   # inside ollama run
API: "options": {"num_ctx": 64000}
ollama ps   # CONTEXT column shows what was allocated
```

llama.cpp:

```bash
llama-server -m model.gguf -c 65536   # -c, --ctx-size; 0 = from model
```

mlx-lm:

```bash
mlx_lm.generate --max-kv-size 65536 ...   # rotating fixed-size KV cache
```

- Note: Ollama: RAM scales with OLLAMA_NUM_PARALLEL × OLLAMA_CONTEXT_LENGTH; keep OLLAMA_NUM_PARALLEL=1 for a single agent.

#### Shrink the KV cache

The cache that holds your context can be quantized too. 8-bit halves its memory with very little quality loss.

ollama:

```bash
OLLAMA_FLASH_ATTENTION=1 OLLAMA_KV_CACHE_TYPE=q8_0 ollama serve
# values: f16 (default), q8_0 (~1/2 memory, recommended), q4_0 (~1/4, more loss at long context); global for all models
```

llama.cpp:

```bash
llama-server -m model.gguf -c 65536 -fa on -ctk q8_0 -ctv q8_0
```

- Note: llama.cpp allowed K/V types: f32, f16, bf16, q8_0, q4_0, q4_1, iq4_nl, q5_0, q5_1 (default f16).

#### Reuse the prompt

Your AGENTS.md and SOUL.md are the same every turn, so cache them instead of re-reading. Keep them stable so the cache keeps hitting.

ollama:

```bash
OLLAMA_KEEP_ALIVE=30m ollama serve   # keep the model loaded between turns
```

llama.cpp:

```bash
llama-server ... --cache-reuse 256   # reuse cached chunks via KV shifting; needs prompt caching
```

mlx-lm:

```bash
cat AGENTS.md | mlx_lm.cache_prompt --model <model> --prompt - --prompt-cache-file agents.safetensors
mlx_lm.generate --prompt-cache-file agents.safetensors --prompt "..."
```

- Note: Editing the top of an always-loaded file invalidates the cached prefix: put changing content at the end.

#### Small model first, big model when needed

Send routine turns to a fast model and hard ones to your biggest. LiteLLM gives both one address.

See: Step 5 litellm.yaml (fast / smart aliases)

- Note: Hermes can also run compression on a separate model via auxiliary.compression (model, base_url).

## Sources

- https://aider.chat/docs/llms/ollama.html
- https://aider.chat/docs/usage/conventions.html
- https://docs.litellm.ai/docs/providers/ollama
- https://docs.ollama.com/context-length
- https://docs.ollama.com/faq
- https://docs.ollama.com/integrations/goose
- https://docs.ollama.com/integrations/hermes
- https://docs.ollama.com/integrations/opencode
- https://docs.openclaw.ai/concepts/agent-workspace
- https://docs.openclaw.ai/concepts/soul
- https://github.com/aaif-goose/goose
- https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md
- https://github.com/ml-explore/mlx-lm
- https://github.com/modelcontextprotocol/servers
- https://goose-docs.ai/docs/getting-started/providers
- https://goose-docs.ai/docs/guides/context-engineering/using-goosehints
- https://hermes-agent.nousresearch.com/docs/user-guide/configuration
- https://hermes-agent.nousresearch.com/docs/user-guide/features/context-files
- https://hermes-agent.nousresearch.com/docs/user-guide/features/personality
- https://hermes-agent.nousresearch.com/docs/user-guide/features/skills
- https://lmstudio.ai/docs/cli
- https://ollama.com/library/gemma4/tags
- https://ollama.com/library/qwen3-coder/tags
- https://ollama.com/library/qwen3.5/tags
- https://opencode.ai/docs/cli/
- https://opencode.ai/docs/config/
- https://opencode.ai/docs/mcp-servers/
- https://opencode.ai/docs/providers/
- https://opencode.ai/docs/rules/