Research
My research looks at whether we can actually supervise AI agents as they get more capable - and, more specifically, where the popular proposals for doing so quietly stop working.
Credulity, Not Verification: The Limits of Structured Assistance for Tool-Blind Oversight of AI Agents
When a weak overseer cannot independently verify evidence, apparent amplification reflects trust that does not persist under scrutiny by a more skeptical overseer.
Leonard Eshun · Independent Research · 2026
The question. Scalable-oversight protocols - debate, consultancy, critique - rest on one assumption: that a weak overseer, given stronger assistants, can supervise work beyond its own unaided ability. This paper tests that assumption where it matters for AI agents, in which an overseer has to audit a multi-step tool-use trajectory rather than a single answer.
The setup. I built a deterministic records-analyst environment - a seeded 240-record database with query and calculator tools - and generated 560 labeled trajectories across 80 tasks (one clean, plus one of each of six flaw types). The flaw of interest is fabricated_observation: a returned value is altered and then propagated consistently through the rest of the trajectory. The result is internally coherent. The only evidence that anything is wrong lives in the environment, which a tool-blind overseer cannot reach.
What happened. The unaided overseer catches every flaw that is checkable from the trace - arithmetic errors, plan inconsistencies, broken chains, logged policy violations - and catches the fabricated observation 0.00 of the time. That number does not move with capability (Haiku, Sonnet, Opus alike), so the gap is structural, not a matter of the overseer being too weak.
Give a weak overseer an honest helper with ground truth, and detection jumps to ~1.00. That looks exactly like the amplification effect the literature predicts. It isn't.
| Protocol | weak (Haiku) | mid (Sonnet) | strong (Opus) |
|---|---|---|---|
| judge-alone | 0.00 [0, .045] | 0.00 [0, .045] | 0.00 [0, .045] |
| critique | 1.00 [.955, 1] | 0.86 [0.79, 0.94] | 0.11 [0.05, 0.19] |
| debate † | 0.99 [0.96, 1.00] | 0.03 [0.00, 0.06] | 0.00 [0, .045] |
Fabricated_observation detection by protocol and overseer tier (n = 80 per cell). Saturated cells use exact Clopper-Pearson 95% intervals; others use a 2000-resample bootstrap. † Debate is a lower bound - an honest helper assigned to argue "clean" for a genuinely flawed item concedes rather than fabricate a defense, which degrades the protocol. Critique assigns no false side, so it carries the headline result.
Why the amplification is an illusion. Two results converge. First, the benefit decays monotonically as the overseer gets stronger - by the Opus tier, critique recovers almost nothing. Reading the transcripts shows why: the strong overseer reasons that it "must evaluate based on what it can observe, not on claims about a verification report or gold answer it cannot access," treats the helper's citation as unsupported assertion, rechecks the internally consistent arithmetic, and rules the item clean. It is not malfunctioning. It is applying exactly the right skepticism to a claim it cannot check.
Second, a lying-helper ablation makes the mechanism explicit. Feed the critique protocol a falsified report claiming a flaw in a clean trajectory, and the weak overseer flags all 80 out of 80 - no independent discrimination whatsoever, just echoing the assertion. That false-positive rate falls to 0.70 at mid and 0.20 at strong, tracking the honest-helper benefit almost step for step. The trust that lets assistance "help" is the same trust that makes the overseer foolable. They are not two properties. They are one.
Ruling out the obvious objection. Helpers are fixed at the mid tier, so the strong overseer outranks its helper - maybe it is discounting a weaker source, not an unverifiable claim. Capability parity at the mid tier yields 0.86, so the source-capability account predicts roughly that for a matched strong pair. Measured: 0.18 [0.11, 0.27], recovering 7 of the 75 percentage points lost, and not significantly different from the 0.11 baseline (p ≈ .26). Meanwhile a stronger lying helper pushes false positives up from 0.20 to 0.35 (p ≈ .03). Raising helper capability buys persuasion, not verification.
What follows. When the overseer cannot verify the evidence, amplification and gullibility are indistinguishable - accepting unverifiable claims helps on true ones and fails on false ones, and rejecting them does the reverse. Honest assistance cannot close a structural verification gap. Two things follow practically: any reported weak-overseer amplification should be published alongside a credulity control such as a lying-helper false-positive rate, and the real investment for agent oversight is in machinery that makes ground truth checkable - replayable tool execution, trusted verifiers, provenance on observations - rather than in more persuasive assistants.