AI Engineering · Part II

The art of AI engineering — harnessing the power of AI for reliable use — Part II

Part I covered what the job of AI engineering actually is, the first judgement call — when to hire the brain — and the hardest part, model and agent evals. This part picks up where the harness begins: how you build the scaffolding around the model, keep it secure and observable in production, and keep improving it over time.

4.Building the scaffolding and harness

The model itself does one narrow thing: it takes a sequence of tokens and returns the next ones. It has no memory of previous conversations, no access to your proprietary data, no way to act on what it knows, and no notion of what it is permitted to touch. Everything that turns this into a working system is the harness.

A harness is the outer layer around a non-deterministic model: the loop that decides when to call it, the tools and data it can reach, the permissions bounding what it can touch, the system prompts and skills that shape its behaviour, the conventions it uses as memory, the orchestration across agent roles, the grading logic that decides whether the work is done, and the recovery logic for when it isn’t.

The field often uses harness and scaffolding interchangeably. However, they behave differently on a model upgrade.

Harness is the load-bearing layer. It exists because the model is a text function, not because it is a weak one: tool interface, permissions and sandboxing, the control loop and its termination conditions, state and recovery, the grading gate.
Scaffolding is the compensatory layer. It exists because this model cannot hold the task alone: prompt patches for a specific failure mode, task decomposition the model can’t do itself, self-verification and retry chains, few-shot examples, output constraints beyond what the schema requires, hand-written planning steps.

Scaffolding will most likely be absorbed into the next model — the history below is largely a record of that happening.

Figure 1. The history of harness evolution — each layer either moved into the harness or was later absorbed back into the model, 2022–2026.

From late 2022. The loop on paper.
ChatGPT in November 2022 had next-token prediction and RLHF, nothing else — no tools, no search, no reasoning. ReAct, published a month earlier, described the reason → act → observe → repeat cycle, but it existed only as a prompting method. Nobody called it a harness.
Spring 2023. Premature autonomy.
AutoGPT and BabyAGI handed the model the whole loop, with tools called by prompt. A loop does not add capability; it amplifies what the model already has, and below a threshold it amplifies errors — 95% per-step reliability across 20 steps is roughly 36% task success. This is the widest the gap ever got between what the harness demanded and what the model could deliver.
Mid 2023. Retrieval bolted on.
RAG gave the model data it did not have: chunk the documents, embed them, retrieve the nearest, stuff them into the prompt. Half of that was harness — the data is not in the weights and never will be. Half was scaffolding for a 4k context window — the chunking, the reranking, the careful choice of what to show. Long context absorbed the second half first; agentic search absorbed the rest, with the model calling grep and search as tools and deciding what to read itself. Retrieval stayed. The retrieval pipeline was the scaffold.
June 2023. Tools enter the API.
Toolformer had shown tool use could be trained in rather than prompted. OpenAI’s function calling made it a product: the model emits a structured call, the harness executes it and returns the result. This is the first harness component to move from prompt to weights.
2023–2024. Retreat to the human in the loop.
Copilot and Cursor closed the gap from the other side: give the human the loop and let the model speed the human up. Devin v1 tried autonomy again; an Answer.AI test put it around 15% success. Meanwhile MCP (November 2024) standardised the tool interface, so the harness stopped being rebuilt per vendor.
Late 2024. The gap inverts.
o1 was the first reasoning model, and for the first time the model curve crossed above the harness curve — the reasoning chain that ReAct had prompted was now in the weights.
February 2025. The curves cross.
Claude Code gave the model bash and file access in the terminal and replaced per-change approval with permission rules — the model got the loop back, and this time it could hold it.
From May 2025. Co-training.
codex-1 was trained with RL on real coding tasks inside environments — the harness became the training environment. Then the absorption started: GPT-5.1-Codex-Max was the first model natively trained to work across context windows through compaction. A harness feature became a model feature.
2026. Shed what got absorbed.
Two measurements fix the size of the harness’s contribution. Harness-Bench ran one model over 106 tasks in different harnesses and scores ranged from 52.4 to 76.2 — a 23.8-point spread with no change to the model. OpenAI tripled GPT-5.6’s ARC-AGI-3 score, 13.3% to 38.3%, by adding only retained reasoning and compaction. And the counter-move: Anthropic deleted 80% of Claude Code’s system prompt once the model no longer needed it. The pace of harness evolution is measured by how much of it you get to delete while keeping the same capability.

So, what does a state-of-the-art harness look like today? The honest answer: it depends on which product you are building on top of. The model is the engine, and the harness is the rest of the car. Nowadays, you can buy the car at four different levels of assembly, and the harness you own is different at each.

Level 1. Buy the whole car. Claude Code, Codex CLI, Gemini CLI. The lab builds the engine and the car together and tunes them to each other. You get the loop, the tools, the permissions and the verifiers out of the box. What you own is the outer ring: instruction files, skills, MCP servers.
Level 2. Buy the lab’s kit. Claude Agent SDK, OpenAI Agents SDK, Google ADK. Same parts as the car above, unbundled so you can assemble them into your own product. The model is still tuned for these parts, so they fit well. You now own context, verification, recovery and observability; the loop and the tool plumbing you still inherit.
Level 3. Buy a generic kit. LangGraph, DeepAgents, Pydantic AI, CrewAI, Mastra. Takes any engine. The frameworks differ mostly in how much of the loop they let you control — LangGraph gives you the graph, CrewAI gives you roles and takes the loop away.
Level 4. Buy the engine only. Open-weight models: Kimi, GLM, DeepSeek, Qwen, MiniMax. The engines have nearly caught up — Kimi K3 leads the Frontend Code Arena ahead of Fable 5 — at a fraction of the price. But nobody built a car around them. You build the whole ring yourself, and you also run the engine yourself: vLLM, quantisation, the tool-call parser. And because the model was not co-trained with your harness, you carry more scaffolding to make it behave. The next open release will absorb some of it — the open labs run the same RL loop — but it absorbs the scaffolding of the harness it was trained against, not yours. The closer your harness sits to the one the lab used, the more you get to delete.

Thomas Wolf from Hugging Face said something similar when people dismissed open-source models as not good enough: stop comparing an engine to a car. The comparison that matters is the trade-off between cost, control and how much harness you want to own — and that depends on the core product you are building on top of.

Figure 2. Anatomy of a harness — the stateless model at the centre, the components that surround it, and the human operator above.

Each component in detail, in build order.

Model call + control loop. Call, read the output, decide whether to call again, stop.
Tool interface. The first tool, with its result fed back into the loop. Now the model can do something. Use bash and the filesystem before any bespoke tools.
Permissions. Before the second tool, not after the first incident. Read-only by default, allowlist per tool and path, sandbox anything that executes. This is also where the human operator enters: what needs approval and what doesn’t.
Observability. Trace every model call and tool call. Too much about how these systems fail is still unknown, so keep every trace — you will need them for debugging now and for review later.
Context management. System prompt, instruction files, retrieval, memory conventions, compaction, indexed knowledge bases, MCP, skills, wikis. Context has always been the bottleneck and probably always will be: the model needs enough of it to do the job, and every token beyond that is paid for in cost and latency. Context management is the discipline of finding that line. It goes hand in hand with observability — the most common failure in production is still a context failure, so you monitor and adjust context in parallel.
Verification. Deterministic checks first: tests pass, the schema is valid, no forbidden files were touched. LLM-as-a-judge second. All of it is defined against the evals we have discussed before.
State and recovery. Checkpoints, retry with the errors in the context, a progress file so a fresh context can pick up where the last one died.
Orchestration. Sub-agents, routing, shared state. Last, and only if one agent has clearly hit its ceiling.

So, three main groups of skills, built in this order:

Runtime — make it act. Model call + control loop, tool interface, permissions.
Feedback — make it see. Observability, context management.
Reliability — make it finish. Verification, state and recovery, orchestration.

Figure 3. The three groups of harness skills, built in order: runtime, feedback, reliability.

And depending on the product you are building on top of, there will be different layers of harness that are built in already or that you have to build or redesign yourself.

Figure 4. Depending on the product you build on top of, some layers come built in and some you have to build or redesign yourself.

Within the scope of this essay I won’t go into the nuts and bolts of designing each layer. There are plenty of resources for that, down to code you can copy and paste — the frontier labs’ own material is the place to start:

Beyond that, think from first principles about your own use case, application and technical limits when designing these layers. If it turns out to be useful, I will add a handbook with code walkthroughs later.

One last thing. Whatever you build in these layers, build it knowing which parts are harness and which are scaffolding. The harness stays. The scaffolding should carry an expiry date, and the next model release is when you check it. In this field, the health of a harness is not measured by how much it does — it is measured by how much of it you got to delete this year while keeping the same capability.

5.Security and observability

A non-deterministic system that reaches for tools and acts on the world needs a new posture on both security and observability — you have to reason about what the model can be talked into, and you have to be able to see what it actually did.

6.CI/CD and continuous innovation

The models keep changing underneath you. Shipping reliably on top of a moving target — and turning each model release into an upgrade rather than a regression — is the last piece of the job.

Part I Back to main