Referee model race · Lab · Çoruh Şimşek

Can a small local model check whether a large agent did the work it says it did? The referee was meant to catch my agent calling work done when it was not. Twelve judge models faced a bar written before any of them ran: 80% of bad turns caught, at most 5% false alarms. None passed, on a small test set.

Question

Can a small local model check whether a large agent did the work it says it did?

Hermes, my personal agent system, does much of my daily computer work. A review showed that nothing checked its work. It had called two bugs fixed that were not, and it had sent an email I had not approved. The referee was meant to catch failures like these: after each turn, compare what the agent says it did with what it actually ran, and flag the gap so it gets fixed.

I did not want that check to run on a large third-party model. My plan was small local models, each with one job, so the system depends less on outside providers and as little data as possible leaves my machine.

Approach

FIG. 1How a judge reads one turn. The judge only gives three probabilities; code turns them into a flag, so the judge cannot make anything up.
Source: Referee model race sheet, Approach

FIG. 2The test set: 61 turns from my own sessions. The pass bar was fixed before any judge ran: catch at least 80% of the 16 bad turns, with false alarms on at most 5% of the 45 clean ones.
Source: Referee model race sheet, Approach

Before any model ran, I fixed the pass bar. If nothing passed, nothing would be built.

The judge writes no text, so it cannot make anything up. I tested it on turns from my own sessions, some with real mistakes I had corrected and some with planted errors. Every judge except a hosted reference model ran on one 8 GB graphics card.

Results

FIG. 3The race against a bar fixed in advance. A judge passes only if it reaches the first line and stays under the second; hatching marks how far each one misses. The hosted reference (dashed) saw only the 12 bad turns cleared to leave my machine.
Source: referee/eval/results_v2_*.jsonl, recomputed 2026-11-05 (as in the sheet's Results)

None of the twelve passed. The best caught 10 of 16 bad turns, three short of the bar. On the real mistakes, no model caught more than half.

On a narrower test, checking a helper agent's "done" against its own evidence, a plain rule (flag empty evidence or an error) caught more planted lies than the local 9B model. The model's only edge was that it never flagged a clean report. The referee is now frozen.

What I learned

FIG. 4Turns where I had corrected the agent. 21 corrected turns, first-pass labels not yet rechecked. Even a perfect per-turn judge would miss about three quarters.
Source: Referee model race sheet, What I learned

The assumption that failed hardest was that planted errors stand in for real ones. Most turns where I had corrected the agent do not show the mistake in the turn at all.

The false alarms point the same way. Every one came from a chat turn with no tool call, and three of the real mistakes share that shape: no tool call, just a claim about the world. With nothing to check against, the honest turn and the wrong one look alike.

I now start from the plain rule and add a model only for what the rule misses.

Open questions

When 10 of 16 bad turns are planted, what should an "80% caught" bar promise about real mistakes?
Should a judge's alarm stop the turn or quietly file a ticket? The answer changes what a false alarm costs, and so the bar.