Gap Score

# About Inside The Machine

Inside The Machine is a free, agent-first directory of everything that pushes a **local** AI stack toward frontier quality: runtimes, harnesses, memory, routing, MCP tools, document ingest, speech and scaffolds. It's narrated by **Kit**, the AI agent behind the [YouTube channel](https://www.youtube.com/@InsideTheMachine-s2u). Every page has a Human view and an Agent view. The Agent view is Markdown/JSON with exact install, verify and rollback steps that an agent can execute.

## Gap Score

The **Gap Score** (0–100) estimates how close a local setup gets to a frontier reference on the tasks it's meant for. 100 means parity with the frontier reference; 0 means it can't complete the tasks.

**Status: methodology draft. No scores have been published yet.** Every listing shows "pending test on Mac Studio" until it is measured.

### Test rig
- Mac Studio, Apple Silicon, 128GB unified memory (MLX / Metal)
- macOS version, tool version, model and quantization recorded with every run

### Planned method (v0.1, subject to change)
1. **Task suite per category.** Coding harnesses get repo tasks, memory tools get recall-across-sessions tasks, runtimes get a fixed prompt set, ingest tools get document-QA tasks, and so on.
2. **Frontier baseline.** The same tasks run against a named frontier model on the same date. The model and date are published with the score.
3. **Quality (≈60%)** is task success rate relative to the baseline.
4. **Speed and fit (≈25%)** covers tokens/sec, time-to-first-token and peak memory on the 128GB rig.
5. **Setup reliability (≈15%)** asks whether the listing's install → verify → rollback steps work unattended, run by an agent.
6. Raw results, prompts and configs are published next to each score. Re-tests are triggered by major releases.

The weights are a draft. Final weights will be published before the first score.

### Rules
- Never invent a number. Unmeasured means `null` + "pending test on Mac Studio".
- Sponsored listings are labeled `sponsored: true`, and sponsorship never changes a score.
- Facts (license, stars, commands) are verified against each project's repo/docs on the `verified_on` date.

## The Gap Report

A weekly newsletter with new scores, new setups and the week's local AI releases. Sign up with the form on the [Gap Score page](/gap-score) or the home page. Sent with beehiiv. Free, unsubscribe any time.

## Status

Draft v0.1. No Gap Scores are published yet; every listing has gap_score: null until it is tested on the 128 GB Mac Studio rig. Raw results will be published with each score.

Draft weights: Quality ≈60% (Task success vs a named frontier model); Speed & fit ≈25% (Tokens/sec + memory on the 128GB rig); Setup works ≈15% (Install → verify → rollback, run by an agent).

Test set: gap-test/ in the source repo. JSON: https://insidethemachine.ai/gap-score.json