OpenAI Evals

Framework and registry for evaluating LLMs.

Measures how close you are to the frontier.

Gap ScorePending test on Mac Studio

20K stars
Other
Updated 2026-04-14

Slow activityLicense unclear

Where it sits

Modelthe weights
Runtimeruns & routes it
Harnessmakes it act
Memorymakes it remember
OpenAI EvalsToolsgives it hands

Catalog entryVerified it exists. Install steps not tested yet.

.md .json

Repo

AGENT VIEW · text/markdown · or send Accept: text/markdown

---
schema: "https://insidethemachine.ai/schema/listing.schema.json"
slug: "openai-evals"
name: "OpenAI Evals"
kind: "repo"
tier: "catalog"
category: "evals"
category_label: "Evals & benchmarks"
stack_layer: "tools"
homepage: null
repo: "https://github.com/openai/evals"
license: "Other"
stars: 19607
last_push: "2026-04-14"
archived: false
tags: ["slow-activity", "license-unclear"]
seen_in_videos: []
gap_score: null
gap_score_status: "pending test on Mac Studio"
verified_on: "2026-10-15"
---

# OpenAI Evals

**What:** Framework and registry for evaluating LLMs.

**Why:** Measures how close you are to the frontier.

**Category:** Evals & benchmarks · **Stack layer:** tools

## Status

Catalog entry. Existence and repo metadata checked 2026-10-15 via the GitHub API. Install commands below are auto-extracted from the README and have NOT been executed or verified. Read the README before running anything; prefer a sandbox or venv.

## Install (unverified)

1. [any] source: README (auto-extracted); status: unverified, not executed

```bash
pip install evals
```

## Verify

No verified check yet. Minimum: the install command exits 0 and the tool's own help/version flag runs (exact flag: see README).

## Rollback (unverified)

1. source: derived inverse of install command

```bash
pip uninstall -y evals
```

## Links

- Repo: https://github.com/openai/evals
- README: https://github.com/openai/evals#readme