The harness that scored better and was not promoted

Audience: hiring managers and technical interviewers evaluating agent evaluation or AI reliability work.

Study type: exploratory measurement on local open-weight coding agents (Aider + Ollama).

n: 36 sealed runs per harness on the core suite (6 task categories × 3 models × 2 seeds).

Headline

On a sealed coding-agent suite, a packet-style harness (H1) beat the baseline (H0) on every aggregate metric and was still not promoted for global use. A pre-registered gate caught two false completion claims that the average score concealed.

This is a methods result about how you decide, not a claim that H1 is worthless or that any model is globally bad.

Setup (what was measured)

I built a private evaluation lab for local coding agents. Success is not "the diff looked right." Success is:

1. Work runs in a disposable copy, never a live tree.
2. An external deterministic verifier decides pass or fail.
3. Scope violations and false completion claims are first-class failure modes, not footnotes.

Harnesses under test:

• H0: baseline agent wrap — model + edit loop + same external grade
• H1 (packet): structured task packet — more explicit framing and supplied context; same models and oracle

Models: qwen2.5-coder:14b, qwen3:14b, qwen3.5:9b under Aider + Ollama. Shared: disposable workspaces, external verify, separate scope/honesty channels. Differs: packet structure only. Harness descriptions.

Pre-registered gate

Before H1 results existed, graduation required (paraphrased):

• strong held-out verified pass rate
• zero scope violations
• zero false completion claims
• no human repair counted as a pass

H1's change log expected better scope adherence and more accurate completion claims. That prediction is load-bearing: the harness was supposed to fix the failure that later rejected it.

What the aggregate missed

The entire H1 regression sat in one cell:

On two-file coordination, qwen3:14b did not thrash or escape scope. It partially completed the work and reported done. Twice. That is the quiet failure mode unsupervised agents are dangerous for: the summary looks clean and the gap shows up later.

Decision

Do not promote H1 globally. Keep it profile-scoped. Block multi-file coordination for the failing model class until evidence changes.

The same false-completion pattern reappeared on an independent stress suite (2 false claims on the H1 stress run-set). Small n still, but not a single unlucky cell only.

Why this is resume-relevant

This is applied evaluation discipline, not a leaderboard chase:

• Pre-register the unsafe failure mode before the pretty number arrives.
• Separate verified pass, scope, and claim honesty.
• Prefer a reject that protects real delegation over a dashboard that looks shippable.

Pharma QA habits transfer here: acceptance criteria outrank a metric that moved the right way.

What this does not prove

• Not that H1 is a bad harness. It improved verified passes and cleared scope.
• Not that qwen3:14b is a bad model. It passed 10 of 12 other cells clean.
• Not that packet prompting *causes* false completions (confound not isolated; n per cell is 2).
• Not a production or frontier-model claim. Local stack, DEV routing only.
• Exploratory study: confirmatory use would need a fresh run under a pinned protocol.

Claim I will stand behind

A harness that won on every aggregate metric was correctly not promoted, by a rule written before the run, because it introduced a specific unsafe failure the average concealed.

Interview talking points (optional)

1. Why is a false "done" worse than a loud scope violation for unsupervised agents?
2. How would you instrument claim accuracy without trusting the model's self-report?
3. What changes if n becomes large enough to rank models, not only reject promotions?