Chat with your files

# Recipe: Chat with your files

A private chat that answers questions about your own PDFs, notes and documents.

URL: https://insidethemachine.ai/do/docs
Memory: needs 16 GB+, best with 32 GB+. Platform: macOS (Apple silicon) first.
Install steps run on an Apple M4 Mac with 16 GB on 2026-10-15. File upload and the first Docling run are checked against the docs only.

## Model by memory (replace {MODEL})

| RAM GB | model | ollama tag | needs GB |
|---|---|---|---|
| 8 | Qwen3.5 4B | qwen3.5:4b | 5.8 |
| 16 | Gemma 4 12B | gemma4:12b | 11.2 |
| 24 | Gemma 4 12B | gemma4:12b | 11.2 |
| 32 | Qwen3.8 27B | qwen3.8:27b | 22.7 |
| 48 | Qwen3.8 27B | qwen3.8:27b | 22.7 |
| 64 | Qwen3.8 27B | qwen3.8:27b | 22.7 |
| 96 | Qwen3.8 27B | qwen3.8:27b | 22.7 |
| 128 | Qwen3.8 27B | qwen3.8:27b | 22.7 |
| 192 | Qwen3.8-Flash-Next (125B-A6B) | qwen3.8-flash-next:125b-a6b | 140 |
| 256 | Qwen3.8-Flash-Next (125B-A6B) | qwen3.8-flash-next:125b-a6b | 140 |
| 512 | Qwen3.8-Flash-Next (125B-A6B) | qwen3.8-flash-next:125b-a6b | 140 |

## Steps

1. **Install Ollama**: Ollama downloads AI models and runs them on your computer. Open the Terminal app and paste this. It may ask for your Mac password.
   ```sh
   curl -fsSL https://ollama.com/install.sh | sh
   ```
   (Or download the app from ollama.com/download)
   Expect: Run `ollama --version`. It prints a version number. If it also says “could not connect”, open Ollama from your Applications folder, wait a few seconds, then go on.
2. **Download your model**: This is the AI itself. It is one big file, so it takes a few minutes.
   ```sh
   ollama pull {MODEL}
   ```
   Expect: The download finishes with “success”.
3. **Add a ChatGPT-style window**: Open WebUI gives you a chat app in your browser. It needs Homebrew (brew.sh) for Python.
   ```sh
   brew install python@3.11
   ```
   ```sh
   python3.11 -m venv ~/.venvs/openwebui && source ~/.venvs/openwebui/bin/activate
   ```
   ```sh
   pip install open-webui
   ```
   ```sh
   open-webui serve
   ```
   (Keep this window open while you use it)
   Expect: Open http://localhost:8080. Make a local account (it stays on your computer) and pick your model at the top.
4. **Add your files**: In a chat, click + and attach a file, then ask about it. For a whole folder, go to Workspace → Knowledge, make a collection and upload your files. Then type # in a chat to use it.

   Expect: The answer quotes your document.
5. **Messy PDF? Clean it first**: Scanned or complex PDFs give bad answers. Docling turns them into clean text you can upload instead.
   ```sh
   python3 -m venv ~/.venvs/docling && source ~/.venvs/docling/bin/activate
   ```
   ```sh
   pip install docling
   ```
   ```sh
   docling yourfile.pdf
   ```
   (Writes yourfile.md next to it)
   Expect: A Markdown file appears. Upload that one.

## Alternative

Prefer one app? Use AnythingLLM. AnythingLLM is a desktop app for chatting with your documents. Download it from anythingllm.com and choose Ollama as the model provider. (https://insidethemachine.ai/setups/anythingllm)

## Rollback

- `ollama rm {MODEL}` (Removes the model file)
- `rm -rf ~/.venvs/openwebui ~/.venvs/docling` (Removes Open WebUI and Docling)
- `rm -rf ~/Library/Caches/pip` (Removes saved downloads)

## One-command install

```bash
curl -fsSL https://insidethemachine.ai/install/docs.sh | bash             # install, then check
curl -fsSL https://insidethemachine.ai/install/docs.sh | bash -s -- plan     # print only
curl -fsSL https://insidethemachine.ai/install/docs.sh | bash -s -- rollback --yes
```

Run status and Windows: https://insidethemachine.ai/install/docs.md. The script picks {MODEL} from the machine's memory (override: ITM_RAM, ITM_MODEL).

## Notes

- Check your free disk space first. You need room for the model plus about 4 GB for Open WebUI and 2 GB for Docling.
- The first Docling run downloads its own small models.
- Answers are only as good as the text it can read. Clean files give better answers.

## If something goes wrong

- “could not connect to ollama app, is it running?”: Ollama is not running. Open Ollama from your Applications folder, or run `ollama serve` in a new Terminal window and leave it open.
- “command not found: ollama”: Close the Terminal window and open a new one. If it still fails, run the install command again.
- “model requires more system memory”: The model is too big for your memory. Pick a smaller memory size at the top of this page and use that model. Close other big apps too.
- “file does not exist” when you download: The model name is wrong, or Ollama is too old. Copy the command again from this page. Then update Ollama: run the install command again.
- It is very slow: Run `ollama ps`. PROCESSOR should say 100% GPU. If it shows CPU, the model does not fit in memory. Pick a smaller model.
- The computer freezes, or the fan is loud: The model uses all your memory. Stop it with `ollama stop {MODEL}`. Then use a smaller one.
- “address already in use” when Open WebUI starts: Another app uses port 8080. Run `open-webui serve --port 8081`, then open http://localhost:8081.
- No models in the list at the top of Open WebUI: Ollama is not running. Start it, then reload the page.
- The answer ignores your file: Check that the file shows in the chat. Scanned PDFs have no real text; clean them with Docling (step 5) and upload the .md file.

## After it works

- Try it: upload a manual and ask “how do I reset it?”. Upload notes and ask “what did we decide?”.
- Make a Knowledge collection per topic, so answers only use the right files.
- See every model you have: `ollama list`.
- Try a different model: copy any `ollama run` name from the Models page.
- Update your model: run `ollama pull {MODEL}` again. It only downloads what changed.
- Free disk space: `ollama rm {MODEL}` deletes a model you no longer use.

## Windows / Linux (not tested by us)

Install Ollama with this PowerShell command, or the installer from ollama.com/download. On Linux, the install command above works as it is. After that, every `ollama` command on this page is the same. Open WebUI and Docling also run on Windows; their pages have the steps.

```powershell
irm https://ollama.com/install.ps1 | iex
```
(Windows: paste in PowerShell. Or get the installer from ollama.com/download)