Human in the Loop: Hallucinations and False Confidence

The problem with models isn’t that they get things wrong. Compilers get things wrong too, and we humans get things wrong all the time. The problem is that they get things wrong in the same tone they get them right.

Confidence is style, not evidence

A model generates plausible text. The certainty of the prose is a property of style, not a measure of truth. “The Array.prototype.findLast method doesn’t exist in Node 16” and “the Array.prototype.findLastIndexOf method takes a second parameter” look exactly equally convincing; one is true and the other is invented on the spot.

We’re socially trained the other way around: we associate confidence with competence. That’s why a model’s false confidence lands so well in the same part of the brain that also believes the consultant who talks loudest.

Where the error actually slips in

Cheap hallucinations —a function that doesn’t exist, a made-up import— die on their own: the compiler or the test kills them in seconds. Nobody reaches production with import { magic } from 'nowhere'.

The expensive ones are another matter. A WHERE that looks correct but ignores the rows with deleted_at. Timezone handling that works in Santiago and breaks on the DST switch. A retry without idempotency that charges twice. They all compile, they all pass the tests the model also wrote, and they’re all plausible on a quick read.

The pattern is clear: the risk isn’t where the machine is obviously ignorant, it’s where it’s subtly confident.

The human doesn’t go “in the loop”, the human goes on the boundary

“Human in the loop” usually gets implemented as “somebody looks at it before merging”, which in practice means somebody approves 800 lines at 6:40pm on a Friday. That isn’t supervision, it’s a formality with a fancy name.

Put the human where the error is irreversible or expensive:

• Money and billing. Always.
• Migrations and deletions. Anything with no undo.
• Personal data and permissions. One WHERE too many here is a legal incident.
• Public contract changes. APIs and events that others consume.

And take the human out of where the machine verifies better: formatting, mechanical refactors, test coverage, dependency updates with a green suite.

Verify instead of believing

The useful question in front of any output isn’t “does this sound right?” but “how do I check it without reading all of it?”. Types, tests, real execution, comparison against a baseline, going to the source. An agent that can run what it wrote and show you the output is qualitatively different from one that only describes it to you.

A practical rule on my team: no claim without attached evidence. If the report says “the tests pass”, the report brings the output. It sounds like bureaucracy until the first time you discover they were never run.

The uncomfortable part

We humans have false confidence too, and we’ve been practicing it longer. The difference is that you calibrate a colleague over time: you know Pedro is optimistic about estimates and that Ana never says “done” without having tested it.

With a model there is no calibration by reputation. Every answer starts from zero, with the same confidence. That’s why the system has to supply the distrust the model doesn’t: not as pessimism, as design.