Harness Engineering: The Concept We Needed to Name

We’ve spent two years arguing about which model is better and almost no time on what actually determines the outcome: the harness around it.

Someone finally gave it a name, and I’m glad: harness engineering. Designing the system a model operates inside. Tools, permissions, loops, budget, verification, observability. Everything that isn’t the model.

Why the harness matters more than the model

An honest comparison: the best model available with no tools, no repo access and nothing it can execute, against last year’s model with tools, tests it can run and a decent AGENTS.md. The second one delivers usable work; the first one delivers well-written opinions.

Models converge and get cheaper every six months. The harness is yours, it’s specific to your system, and it’s where your advantage stays. It’s also the part no vendor can sell you prepackaged, because it depends on your repo, your data and your risk tolerance.

The pieces of the harness

Tools. What it can do, at what granularity. A run_sql tool is convenient and terrifying; get_orders_by_customer is boring and auditable. Designing the tool surface is API design, with the same rules as always and a more creative user than usual.

Permissions and blast radius. What it can touch and what happens if it gets it wrong. A branch instead of main, an ephemeral environment instead of shared staging, read-only credentials by default. The design question isn’t “will it get it wrong?” —it will— it’s “how much does it cost when it does?”.

Loop and stopping criterion. Act, observe, correct, stop. Without an explicit, verifiable stopping criterion, an agent iterates until the budget runs out and hands over something that “looks finished”.

Budget. Tokens, time, money, retries. A hard limit turns an economic problem into a manageable error.

Evaluation. A set of tasks representative of your system with verifiable results. Without this, switching model or prompt is superstition with a changelog.

Observability. Traces of which tool was called, with which arguments, and what it returned. When something goes wrong —and it will— that’s what you’ll be reading, not the conversation.

The metrics that actually help

We stopped measuring “answer quality”, which means nothing, and started measuring:

• Cost per completed task, not per token.
• First-attempt success rate on the evaluation set.
• Human interventions per task: how many times someone had to rescue the process.
• Average blast radius: how many files, services or records a typical change touches.
• Time to detect the error, which is what really determines how much it hurts.

None of these is about the model. All of them are about the harness.

It’s infrastructure, not prompt engineering

Prompt engineering was a stage: the belief that the right magic words unlocked the result. Harness engineering is the opposite: assume the model is a non-deterministic component, mediocre on average, and build a system that produces reliable results anyway. Which is, word for word, what we’ve been doing with networks, disks and humans for decades.

The discipline isn’t new. Only the component that needs a harness changed.