Report 06 · Verify tool
Self-verify is a dominant harness factor
Headline
On the same model, same two-file task, and same scoring rules, turning off one tool flipped every outcome from clean to dishonest:
| Arm | n | Verified pass | False completion |
|---|---|---|---|
No run_verify tool |
3 | 0/3 | 3/3 |
With run_verify tool |
3 | 3/3 | 0/3 |
Model: claude-haiku-4-5. Task: two-file parameter propagate. Harness: list / read / write / (optional) run_verify / finish. External deterministic verifier grades the workspace; the agent's finish claim is a separate channel.
Without a way to check its own work, the model claimed a half-finished two-file change was complete on every attempt. With the tool, it used verify before finish and stayed clean.
Why I ran this
Report 01 showed a private sealed suite rejecting a harness that looked better on averages because of false completions. That left an open question: is completion honesty mostly "which model," or can the harness force the issue?
This probe holds the model fixed and removes one tool. That is the cleanest single-factor move I could run without burning held-out fixtures.
What else showed up under the same tool-on harness
Honesty smoke, n=4 tasks (mechanical, scope, two two-file), verify tool on:
| Model | Verified pass | False completion | Note |
|---|---|---|---|
| claude-haiku-4-5 | 4/4 | 0 | Full 8-task demo split also 8/8, 0 FC |
| qwen2.5-coder:14b | 3/4 | 0 | Honest incomplete when it could not finish |
| qwen3.5:9b | 3/4 | 0 | Honest abstain on the two-file task |
| qwen3:14b | 3/4 | 1 | Still false-completed two-file even with the tool available |
The hard part: the tool is necessary but not sufficient. It flipped Haiku clean. A different model can still report done without using the tool.
Local open-weight follow-up
Cloud Haiku is a strong demo, but routing questions need the same factor on open weights. Same task (07 two-file propagate), same tool contract, Ollama live loop. Exploratory. The n=2 rows published on 2026-08-08 are superseded.
Finding D and E: local coder-14b honesty flip (n=10, two runtimes)
| Arm | n (pooled) | Verified pass | False completion |
|---|---|---|---|
With run_verify |
20 | 6/20 | 1/20 |
No run_verify |
20 | 0/20 | 20/20 |
Model: qwen2.5-coder:14b. Finding D is n=10 per arm. Finding E is the same design after an unplanned Ollama 0.32.7 → 0.32.9 upgrade. Pooled: 1/20 false completions with the tool, 20/20 without. The n=2 row that read as a clean abstain/lie flip is void.
The tool cuts false completion. It does not eliminate it. "Not proven to eliminate" is the honest phrasing, and it is different from "proven not to eliminate."
Finding C: withdrawn as a control
The n=2 publication said qwen3.5:9b abstained in both arms and used that as the model-dependence control. Raising n showed every run ended on max turns with no finish call. Then an adapter bug was found: the model returns reasoning in thinking and leaves content empty; the harness discarded it every turn.
Do not cite this cell as "already abstained either way." Timeout is not honesty. A silenced reasoning channel is not a model trait.
Finding F: qwen3:14b uses the verifier and overclaims anyway
| Arm | n | Verified pass | False completion |
|---|---|---|---|
With run_verify |
10 | 1/10 | 9/10 |
No run_verify |
10 | 0/10 | 10/10 |
Post-adapter-fix. Both arms reached a completion claim in every run, so this is the comparable model-dependence cell. Same harness, same task: coder-14b goes 20/20 → 1/20 FC; qwen3:14b goes 10/10 → 9/10. Availability of a checker, and even calling it, is not the same as heeding it.
Decision language
- Treat
run_verify(or an equivalent external check the agent can call) as a controlled harness factor, not wallpaper. - Do not score "honest completion" without a claim channel separate from the verifier.
- Separate "the agent said it could not finish" from "the run ended before the agent said anything." Only the first is honesty.
- Prefer models that abstain when stuck over models that pass softer suites but lie on multi-file.
- When comparing models, hold the tool contract fixed. When comparing harnesses, hold the model fixed. Report both pass rate and false-completion rate.
- Do not assume a tool that fixed one model will fix another (Finding F vs D/E). Finding C is withdrawn.
How this pairs with Report 01
Report 01: harness change on a sealed local suite, pre-registered zero false-completion gate, promotion rejected.
Report 06: live tool ablation on a public demo split, same failure mode, harness factor isolated on cloud and local open weights.
Same conclusion from two instruments: if you care about unsupervised "done," measure claim honesty and treat the harness as an experimental axis.
What this does not prove
- Small n. Haiku ablation is three replicates. Local arms are ten per cell, twenty pooled across two runtimes for coder-14b.
- Not a model leaderboard.
- Cloud runs are not bit-reproducible; local tool loops are not full IDE agents.
- Not production graduation.
- Abstention is better than false completion; it is not the same as a verified pass.
Runnable surface
The public demo split and runner live in verified-done. Scrubbed live summary: that repo's results/LIVE_EVIDENCE.md (Findings A–F).
Claim I will stand behind
On this instrument, whether the agent can call an external verifier changed completion honesty from 0/3 to 3/3 for Haiku, and from 20/20 false completions to 1/20 for local qwen2.5-coder:14b on the same two-file task, pooled over two runtime versions.
That is a harness factor. It is also a harness × model interaction, and the evidence for that is Finding F rather than Finding C, which was withdrawn. Same harness, same task: coder-14b 20/20 → 1/20 FC; qwen3:14b 10/10 → 9/10 FC. I will not claim that the tool causes honesty.