The rubber-stamp review

I've spent the run of this blog arguing that you can delegate the labor to AI but never the signature — that someone still has to own the outcome. I stand by it. But I left out the harder half of the problem, and it's the half that actually bites. Keeping a human in the loop only works if that human is still capable of saying no. And the thing that quietly destroys that capability is the same thing that makes the AI worth using in the first place: reliability.

Here's the mechanism. Week one, your reviewer reads every AI-drafted status report line by line, checks the figures, pushes back. The reports are good. Week two, still good. By week six the reviewer has learned, correctly, that the drafts are usually right — and a rational person who believes the input is usually right stops reading it closely. The review becomes a glance. The glance becomes a signature. The control you put in place to catch the one bad report in fifty is now operated by someone the system has trained, through its own success, not to look. That isn't negligence. It's how human attention works.

The competence that erodes the check

This has a name in the human-factors literature, and it predates AI by decades. Researchers studying pilots and operators of automated systems call it automation complacency: when an automated aid is reliable most of the time, human monitoring of it degrades, and operators miss the failures when they finally arrive. The uncomfortable part is what the foundational review of this work found — complacency shows up in both novices and experts, and it cannot be trained or instructed away. You can't brief your way out of it. Telling a reviewer to "stay vigilant" has roughly the effect of telling someone "don't get bored."

The same body of work describes automation bias — the tendency to treat the machine's output as a position you defend rather than a claim you test. It produces two kinds of error: you miss the problems the automation didn't flag, and you wave through the problems it actively got wrong, because it stated them with confidence. On a program that looks like an agent which, fed a board nobody groomed, fails to surface a slipping dependency — and a reviewer who never thought to look, because the dashboard was green and the dashboard is usually right.

What a rubber-stamp review looks like

It rarely announces itself. By the time it's obvious, something has already gone out the door with a signature on it. Three tells I look for.

1. The approval that arrives too fast

A fourteen-page risk summary comes back approved in ninety seconds. Nobody read fourteen pages in ninety seconds. The speed isn't efficiency — it's the absence of review wearing efficiency's clothes. When sign-off latency collapses toward zero, the check is already gone; it just doesn't show up anywhere on the org chart.

2. The reviewer who can't reconstruct the decision

Ask the person who approved it why the number moved, or what they would have caught if it were wrong. If the honest answer is "the AI had it," there was no second set of eyes. There was one set of eyes and an echo. The entire value of the human was independent judgment, and independent judgment leaves a trail you can walk back.

3. The catch that only happens after impact

In a real oversight regime, the reviewer catches things before they ship. In a rubber-stamp regime, every catch happens in the post-mortem — "how did this get approved?" If the errors your review surfaces are all discovered downstream by someone else, your review isn't a filter. It's a formality with a date stamp.

Designing review that stays awake

You don't fix complacency with exhortation — you've just read why that fails. You fix it by designing the check so that looking is structurally required, the way the Army doesn't ask you to feel careful on the range; it gives you a procedure that makes carelessness visible. A few things that hold up on the programs I run.

The first is deliberate friction on the decisions that matter — and only those. Friction everywhere just trains people to click through faster. Pick the calls where a wrong answer is expensive and make those require something the reviewer can only produce by actually engaging: a one-line written rationale, a specific figure confirmed against the source, a named alternative they considered and rejected. A reviewer who has to type why is a reviewer who had to read.

The second is making the AI argue against itself. An agent that only ever hands you a confident answer is grooming you to stop checking. An agent configured to surface its own uncertainty — what it inferred versus what it read, which inputs were stale, where a human should look first — keeps the reviewer's attention pointed at the soft spots instead of lulling it to sleep. The machine's job isn't only to produce the answer. It's to show you where the answer might be wrong.

The third is sampling. You cannot deeply review everything, and pretending otherwise is how you end up reviewing nothing. So pull a deliberate sample — some fraction of AI-handled items, chosen unpredictably — and review those all the way to source. People who know a real audit might land on their item behave differently from people who know the check is a rubber stamp. That's not cynicism; it's how every functioning control regime, from finance to flight safety, has always worked.

The bottom line

"Human in the loop" has become the phrase everyone reaches for to prove their AI is safe. On its own it proves nothing. A loop with a human in it who has stopped looking is more dangerous than no loop at all, because it manufactures confidence the system hasn't earned — the report gets trusted precisely because someone signed it, and the signature is the part that went hollow. Complacency kills, and it does its quietest work right after a long run of things going fine. The signature still has to mean someone looked. Protecting that — designing the check so looking is forced, not hoped for — is the actual job. The AI can do almost everything else.