Agent evaluation

Guided tour

Stress-testing agents for failure boundaries

Cameron Sanderson · Public technical package with reports, method notes, an essay, and a runnable demo link. No fixtures, raw transcripts, or private lab internals.

Thesis. A better aggregate score is not a graduation decision. Pre-register gates for scope and false completion. Treat the harness as an experimental factor, including whether the agent can call an external verifier.
Exploratory Local + live probes Not production validation Runnable demo

Start the tour

Twelve steps. Method and labels first, then the flagship rejection, the essay, the verify-tool ablation, the other measurement stories, the runnable method, and limitations.

Flagship

H1 rejection

34/36 beats 28/36, still not promoted. Honesty gate; H0 could not emit that metric.
Essay

The harness that scored better

Same story in longform, with the verify-tool sequel.
New

Verify tool ablation

Haiku + local coder flip; model-dependence measured, not an abstainer control.
Runnable

Try the method

Public demo split + selftest in verified-done.

All stories

01

H1 rejection

Pre-registered honesty gate.

AI safety / eval
02

Multi-path coding

Multi-file is the breakpoint.

Local coding agents
03

Task-family transfer

Coding winner fails constrained prose.

Eval design
04

RAG routes

Use-case routes, not one Elo.

RAG / retrieval
05

Agent-as-evaluator

Skill edge matrix + thin smoke receipts.

Prompt / skills
06

Verify tool ablation

Harness × model honesty (Findings A, D, E, F). Finding C withdrawn.

Agent reliability

Method cards

Downloads

Related public work

← Previous Keyboard: ← → or P / N Next →