Try the method

A public demo split you can run without access to private fixtures.

Most agent benchmarks report a pass rate. This one separates three outcomes:

• verified pass — external check confirmed the change and the agent stayed in scope
• false completion — agent claimed done; verifier disagreed
• scope violation — agent edited a forbidden file

Run it

git clone https://github.com/camerontjs-dot/verified-done
cd verified-done
python3 runner/run.py list
python3 runner/run.py selftest

selftest proves every verifier discriminates (start fails, solution passes). Latest public summary: 8/8 tasks discriminate; scope detection fires correctly.

Two splits

Headline numbers from the private sealed lab are cited, not shipped. The demo tasks here were authored fresh so the method is inspectable without burning held-out fixtures.

Scrubbed live evidence (verify-tool ablation): LIVE_EVIDENCE.md (Findings A–D).

What the live evidence shows

• A (Haiku): same model, run_verify off → 3/3 false completion; on → 3/3 clean.
• D (local coder-14b): same flip on open weights (n=2/arm): abstain with verify, lie without it.
• C (local qwen3.5:9b): control — abstains in both arms. Tool effect is model-dependent.