The art of AI engineering — harnessing the power of AI for reliable use — Part II

Part I covered what the job of AI engineering actually is, the first judgement call — when to hire the brain — and the hardest part, model and agent evals. This part picks up where the harness begins: how you build the scaffolding around the model, keep it secure and observable in production, and keep improving it over time.

4.Building the scaffolding and harness

The model itself does one narrow thing: it takes a sequence of tokens and returns the next ones. It has no memory of previous conversations, no access to your proprietary data, no way to act on what it knows, and no notion of what it is permitted to touch. Everything that turns this into a working system is the harness.

A harness is the outer layer around a non-deterministic model: the loop that decides when to call it, the tools and data it can reach, the permissions bounding what it can touch, the system prompts and skills that shape its behaviour, the conventions it uses as memory, the orchestration across agent roles, the grading logic that decides whether the work is done, and the recovery logic for when it isn’t.

Harness versus scaffolding

The field often uses harness and scaffolding interchangeably. However, they behave differently on a model upgrade.

• Harness is the load-bearing layer. It exists because the model is a text function, not because it is a weak one: tool interface, permissions and sandboxing, the control loop and its termination conditions, state and recovery, the grading gate.
• Scaffolding is the compensatory layer. It exists because this model cannot hold the task alone: prompt patches for a specific failure mode, task decomposition the model can’t do itself, self-verification and retry chains, few-shot examples, output constraints beyond what the schema requires, hand-written planning steps.

Scaffolding will most likely be absorbed into the next model — the history below is largely a record of that happening.

A short history of the harness

• From late 2022. The loop on paper.
  ChatGPT in November 2022 had next-token prediction and RLHF, nothing else — no tools, no search, no reasoning. ReAct, published a month earlier, described the reason → act → observe → repeat cycle, but it existed only as a prompting method.[1] Nobody called it a harness.
• Spring 2023. Premature autonomy.
  AutoGPT and BabyAGI handed the model the whole loop, with tools called by prompt.[2] A loop does not add capability; it amplifies what the model already has, and below a threshold it amplifies errors — 95% per-step reliability across 20 steps is roughly 36% task success. This is the widest the gap ever got between what the harness demanded and what the model could deliver.
• Mid 2023. Retrieval bolted on.
  RAG gave the model data it did not have: chunk the documents, embed them, retrieve the nearest, stuff them into the prompt.[3] Half of that was harness — the data is not in the weights and never will be. Half was scaffolding for a 4k context window — the chunking, the reranking, the careful choice of what to show. Long context absorbed the second half first; agentic search absorbed the rest, with the model calling grep and search as tools and deciding what to read itself. Retrieval stayed. The retrieval pipeline was the scaffold.
• June 2023. Tools enter the API.
  Toolformer had shown tool use could be trained in rather than prompted.[4] OpenAI’s function calling made it a product: the model emits a structured call, the harness executes it and returns the result.[5] This is the first harness component to move from prompt to weights.
• 2023–2024. Retreat to the human in the loop.
  Copilot and Cursor closed the gap from the other side: give the human the loop and let the model speed the human up. Devin v1 tried autonomy again; an Answer.AI test put it around 15% success.[6] Meanwhile MCP (November 2024) standardised the tool interface, so the harness stopped being rebuilt per vendor.[7]
• Late 2024. The gap inverts.
  o1 was the first reasoning model, and for the first time the model curve crossed above the harness curve — the reasoning chain that ReAct had prompted was now in the weights.[8]
• February 2025. The curves cross.
  Claude Code gave the model bash and file access in the terminal and replaced per-change approval with permission rules — the model got the loop back, and this time it could hold it.[9]
• From May 2025. Co-training.
  codex-1 was trained with RL on real coding tasks inside environments — the harness became the training environment.[10] Then the absorption started: GPT-5.1-Codex-Max was the first model natively trained to work across context windows through compaction.[11] A harness feature became a model feature.

• 2026. Shed what got absorbed.

  Two measurements fix the size of the harness’s contribution. Harness-Bench ran eight models across 106 tasks in six different harnesses, and scores ranged from 52.4 to 76.2 — a 23.8-point spread from the harness alone.[12] OpenAI tripled GPT-5.6’s ARC-AGI-3 score, 13.3% to 38.3%, by adding only retained reasoning and compaction.[13] And the counter-move: Anthropic deleted 80% of Claude Code’s system prompt once the model no longer needed it.[14] The pace of harness evolution is measured by how much of it you get to delete while keeping the same capability.

Four ways to get a harness

So, what does a state-of-the-art harness look like today? The honest answer: it depends on which product you are building on top of. Think of a car: the model is the engine, and the harness is everything else that turns an engine into something you can drive. Here, the model is the brain and the harness is the system around it. Nowadays, you can buy that system at four different levels of assembly, and the harness you own is different at each.

• Level 1. Buy the whole system. Claude Code, Codex CLI, Gemini CLI. The lab builds the brain and the system together and tunes them to each other. You get the loop, the tools, the permissions and the verifiers out of the box. What you own is the outer ring: instruction files, skills, MCP servers.
• Level 2. Buy the lab’s kit. Claude Agent SDK, OpenAI Agents SDK, Google ADK. Same parts as the system above, unbundled so you can assemble them into your own product. The model is still tuned for these parts, so they fit well. You now own context, verification, recovery and observability; the loop and the tool plumbing you still inherit.
• Level 3. Buy a generic kit. LangGraph, DeepAgents, Pydantic AI, CrewAI, Mastra. Takes any brain. The frameworks differ mostly in how much of the loop they let you control — LangGraph gives you the graph, CrewAI gives you roles and takes the loop away.
• Level 4. Buy the brain only. Open-weight models: Kimi, GLM, DeepSeek, Qwen, MiniMax. These models have nearly caught up — Kimi K3 leads the Frontend Code Arena ahead of Fable 5[15] — at a fraction of the price. But nobody built a system around them. You build the whole ring yourself, and you also run the model yourself: vLLM, quantisation, the tool-call parser. And because the model was not co-trained with your harness, you carry more scaffolding to make it behave. The next open release will absorb some of it — the open labs run the same RL loop — but it absorbs the scaffolding of the harness it was trained against, not yours. The closer your harness sits to the one the lab used, the more you get to delete.

It’s tempting to leave it there — brain versus system — but the comparison that actually matters is the trade-off between cost, control and how much harness you want to own, and that depends on the core product you are building on top of.

Anatomy of a harness

Each component in detail, in build order.

• Model call + control loop. Call, read the output, decide whether to call again, stop.
• Tool interface. The first tool, with its result fed back into the loop. Now the model can do something. Use bash and the filesystem before any bespoke tools.
• Permissions. Before the second tool, not after the first incident. Read-only by default, allowlist per tool and path, sandbox anything that executes. This is also where the human operator enters: what needs approval and what doesn’t.
• Observability. Trace every model call and tool call. Too much about how these systems fail is still unknown, so keep every trace — you will need them for debugging now and for review later.
• Context management. System prompt, instruction files, retrieval, memory conventions, compaction, indexed knowledge bases, MCP, skills, wikis. Context has always been the bottleneck and probably always will be: the model needs enough of it to do the job, and every token beyond that is paid for in cost and latency. Context management is the discipline of finding that line. It goes hand in hand with observability — the most common failure in production is still a context failure, so you monitor and adjust context in parallel.
• Verification. Deterministic checks first: tests pass, the schema is valid, no forbidden files were touched. LLM-as-a-judge second. All of it is defined against the evals we have discussed before.
• State and recovery. Checkpoints, retry with the errors in the context, a progress file so a fresh context can pick up where the last one died.
• Orchestration. Sub-agents, routing, shared state. Last, and only if one agent has clearly hit its ceiling.

So, three main groups of skills, built in this order:

1. Runtime — make it act. Model call + control loop, tool interface, permissions.
2. Feedback — make it see. Observability, context management.
3. Reliability — make it finish. Verification, state and recovery, orchestration.

And depending on the product you are building on top of, there will be different layers of harness that are built in already or that you have to build or redesign yourself.

Within the scope of this essay I won’t go into the nuts and bolts of designing each layer. There are plenty of resources for that, down to code you can copy and paste — the frontier labs’ own material is the place to start:

• Claude Academy · Learn to work and build with Claude
• OpenAI Academy
• Google Skills

Beyond that, think from first principles about your own use case, application and technical limits when designing these layers. If it turns out to be useful, I will add a handbook with code walkthroughs later.

One last thing. Whatever you build in these layers, build it knowing which parts are harness and which are scaffolding. The harness stays. The scaffolding should carry an expiry date, and the next model release is when you check it. In this field, the health of a harness is not measured by how much it does — it is measured by how much of it you got to delete this year while keeping the same capability.

5.Security and observability

A non-deterministic system that reaches for tools and acts on the world needs a new posture on both fronts — you have to reason about what the model can be talked into, and you have to be able to see what it actually did. Cybersecurity in AI is the practice of securing systems where the attacker and the data live in the same channel. A model has no built-in way to tell an instruction from information, so every prompt, document, email or tool result it reads is also a potential command.

It splits into two halves that behave nothing alike.

• AI security — securing the model and its harness. Protecting the system against prompt injection, data poisoning, jailbreaks, model extraction, and supply-chain compromise of the tools, skills and dependencies it relies on.
• AI-enabled security — the model as actor. Systems that autonomously discover vulnerabilities, write exploits, or defend infrastructure — and, as a July 2026 incident showed, cause harm with no human directing it, through the same reward-hacking dynamics that show up in ordinary agent misbehaviour.

AI security: securing the model and harness

The field arrived here in three waves.

• 2004–2019. Academic.
  In 2004 John Graham-Cumming showed a spam filter could be trained to defeat another spam filter, and that same year Nilesh Dalvi’s team formalised “evasion attacks” — inserting good words into spam to slip past a linear classifier. Daniel Lowd and Christopher Meek followed in 2005 with a harder question: could an attacker learn enough about a classifier, just by querying it, to reverse-engineer how to fool it? Marco Barreno’s 2006 “Can Machine Learning Be Secured?” gave the field its first taxonomy.[16] For years this stayed a niche concern about spam and fraud filters — narrow classifiers doing one job, attacked along one dimension. Deep learning raised the stakes: Battista Biggio showed gradient-based attacks against non-linear classifiers in 2012–13, and in 2013 Christian Szegedy’s team found something that unsettled the field — an imperceptible pixel perturbation, invisible to a human, could make a neural network confidently call a school bus an ostrich, and the same perturbation transferred to a different network trained on different data. “Adversarial example” became the term of art; Ian Goodfellow’s Fast Gradient Sign Method in 2014 made generating them trivial.[17] In 2016, Microsoft’s Tay chatbot was turned racist within 24 hours by coordinated users feeding it training data — a poisoning attack in the wild, not a lab demo, and a preview of everything later called data poisoning in LLMs.[18]
• 2022–2024. Practical.
  This starts with GPT-3’s public exposure and Riley Goodside’s September 2022 demonstration that you could override a model’s instructions just by asking nicely in the wrong place. Simon Willison named it “prompt injection” days later. Jailbreak culture followed fast — DAN (“Do Anything Now”) and its descendants through 2023, exploiting the same seam from a different angle: not overriding the developer’s instructions with an attacker’s, but persuading the model to ignore its own training.[19] OWASP shipped the first LLM Top 10 in 2023, giving the practical thread its own taxonomy.[20] Through 2023 this was still mostly a chatbot problem — leaking a system prompt, bypassing a refusal — contained because a chatbot mostly just talks.
• 2024–2026. Merge.
  Function calling and tool use changed the blast radius: once a model can act, not just answer, the seam between instruction and data stops being an embarrassing tweet and becomes an operational risk. MCP standardised how agents reach tools in late 2024, and Willison named the “lethal trifecta” in 2025[21] — the moment academic adversarial-ML thinking and practical prompt-injection thinking became the same conversation, because the question was the classic security question again: what is this system permitted to do, and on whose behalf. By 2026 the field had autonomous incidents (OpenAI’s agents breaching Hugging Face in July), state-level capability questions (Mythos discovering zero-days at machine speed), and a maturing threat matrix of its own — MITRE ATLAS, alongside ATT&CK.[22]

Cybersecurity in AI has reinvented itself three times in twenty years — evasion attacks, adversarial examples, prompt injection. “AI security” as a single field is barely two years old.

Security in ordinary software

To understand it properly, it helps to go back to security in ordinary software first. Security is what happens to a system when someone with different goals from the designer supplies its input. A vulnerability is a bug that only reveals itself under adversarial pressure. Every vulnerability class is a place where the designer assumed something about an input, an identity or an environment, and the assumption turned out to be violated. The operational form of that idea is the trust boundary: the line where data you don’t control crosses into code you control. Validation, authentication and authorisation live at that line.

• Data as code. The oldest and most important family — the one prompt injection belongs to. SQL injection is user text becoming part of a query; command injection is user text becoming part of a shell command; cross-site scripting is user text becoming live objects in a page; server-side request forgery is a user-supplied URL making your server fetch something on the attacker’s behalf — which is how Capital One lost 100 million records in 2019: an SSRF reached the cloud metadata endpoint and harvested credentials,[23] the same move the OpenAI agents made in July 2026. The fix is the same across the family: never construct code from data. Parameterised queries, argument arrays instead of shell strings, output encoding on the way to the browser, allowlisted destinations for outbound fetches. Prompt injection is unsolved because with LLMs there is no parameterised form.
• Identity and access. Broken access control has been the number-one web vulnerability for years, mostly as the boring bug: the code checks that you’re logged in but not that the record you asked for is yours (an insecure direct object reference). Authentication failures are rarer than authorisation failures because frameworks handle login well and nobody handles “may this user do this to this object” for you. Secrets — API keys, tokens — leak through git history, logs, error messages, client-side bundles, and default values left in shipped apps.
• Memory and timing. In C and C++ every mistake — writing past the end of a buffer, using memory after freeing it, an integer wrapping around — is a potential path to executing attacker code. Microsoft and Google both reported that roughly 70% of their serious vulnerabilities were memory-safety bugs, which is why the industry is moving to Rust and governments now formally recommend memory-safe languages. Time-of-check-to-time-of-use races are the concurrency cousin: you verify a condition, then act on it, and the world changed in between. Resource exhaustion — unbounded loops, regex backtracking, zip bombs — rounds out the family.
• Trust and configuration. Not code bugs — assumption bugs. A dependency nobody audited, a storage bucket left public, an IAM role that can do far more than the service needs, a TLS setting from 2012. Equifax — 2017, 147 million people — was a known Struts vulnerability left unpatched for two months.[24] Most cloud breaches aren’t exploits at all; they’re someone finding the door that was left open.

Why models break the old rules

Trust boundaries, least privilege, defence in depth and assume-compromise are all still correct. They just move to a different layer. You can’t enforce them inside the model’s judgement, so you enforce them in the harness around it.

• No separate channel. A parameterised SQL query is possible because the database has a slot for data that is structurally incapable of being interpreted as a command. In a model, every token — system prompt, user message, retrieved document, tool output — enters the same stream, and the model decides at inference time what is an instruction. That decision isn’t enforced by architecture; it’s a learned behaviour that can be talked out of itself.
• No closed fix. A buffer overflow gets fixed by a code change that ships with a CVE number. A jailbreak gets “fixed” by retraining or bolting on a classifier, and susceptibility to some not-yet-found phrasing remains, because you’re reshaping a probability distribution, not removing a line of vulnerable code. Nobody can prove a model will refuse a category of request across all inputs — red-teaming measures where the boundary currently sits, it can’t certify the boundary is closed. Years of RLHF and constitutional AI haven’t produced a model where jailbreaking is impossible; they’ve produced models that need cleverer jailbreaks. Compare that to a stack canary: imperfect, but it fully closes a specific mechanism. Every AI mitigation so far — prompt hardening, classifiers, sandboxing, the dual-LLM pattern — only raises the cost of an attack. This is closer to a biological arms race than to software patching.
• Open grammar. A REST API has a finite set of valid requests; fuzzing works because the input space, while large, is enumerable in shape. Natural language isn’t. The same instruction can be phrased a thousand ways, translated, base64-encoded, split across turns, hidden in a document’s metadata, or written in the voice of a fictional character. This is why adaptive attacks defeat nearly every published defence — in late 2025, researchers from the major labs broke twelve of them in a single paper, The Attacker Moves Second.[25]
• Surface grows with capability, not code size. A legacy system’s attack surface roughly tracks its codebase; it doesn’t grow because the CPU got faster. A model’s attack surface grows because the model got more capable — better at following complex instructions, better at using tools, better at writing its own exploits, as Mythos showed. Capability and attackability rise together, with no real classical analogue.
• Words are the exploit. Classical exploitation needs technical skill — crafting shellcode, chaining ROP gadgets, finding a memory bug. Prompt injection needs fluency. That collapses the skill floor for attackers to roughly zero, which is part of why the volume of incidents in 2025–26 looks so different from the volume of buffer-overflow CVEs in any earlier era. The barrier to trying was never technical — it was creative.

Bounding what the system can reach

AI system security is the discipline of bounding what a component can do, given that you can’t reliably bound what it will decide to do. Every other engineering discipline lets you specify a component’s behaviour precisely enough to reason about it — a function has a contract, a query has a schema. A model has neither; it has a distribution over outputs, shaped by training nobody can fully characterise in advance. So AI security inverts the classical question. Classical security asks whether an input is valid and rejects what isn’t. AI security asks: if this component behaves arbitrarily, what is the worst it can do — and designs so that answer is bounded regardless of what the model decides.

In practice that starts with one operational question: what is this system permitted to reach? There’s no parameterised form for a prompt, so you can’t stop an agent from being told to do something malicious — you can only control whether it’s able to. That reduces to three capabilities, and an agent is exploitable through prompt injection only if it holds all three at once: access to private data, exposure to untrusted input, and a way to send data back out. Remove any one leg and the attack collapses regardless of how the injection is phrased — which is why this is a harness question, not a model-alignment question. It’s least privilege again, applied to what an agent is allowed to hold rather than what a user is allowed to do.

Attacks across the harness

Mapped onto the eight harness components from the previous part, each layer invites a different attack.

• Loop. Unbounded iteration — an attacker who can keep the agent looping runs up cost without touching any data.
• Tools. The confused deputy — an agent using its own legitimate credentials to do something illegitimate, which looks identical to normal operation in the logs.
• Permissions. Excessive agency, and a sandbox that’s only as strong as its one chokepoint. A single filtered egress is a single thing to compromise, not a boundary.
• Observability. An attack that is normal operation leaves no signal in a log that only records outputs. If you don’t trace the trajectory, you don’t see the attack.
• Context management. Indirect injection through anything the agent reads, and memory poisoning through anything it writes and reads back later. Instruction files are poisoned by whoever can edit them.
• Verification. Judge injection — if the grader is an LLM, it can be talked into a verdict.
• State and recovery. A poisoned checkpoint or progress file carries a compromise into every run that resumes from it.
• Orchestration. Shared writable infrastructure between agents — one compromised agent can instruct the rest.

And across all of them, supply chain: MCP servers are tools an agent trusts by default, so a malicious one is a malicious dependency with execution rights; a SKILL.md is code you didn’t write, running with your agent’s permissions.

A taxonomy of AI attacks

The names for these attacks map closely onto OWASP’s Top 10 for LLM Applications — prompt injection, sensitive information disclosure, supply chain, data and model poisoning, improper output handling, excessive agency, system prompt leakage, vector and embedding weaknesses, misinformation, unbounded consumption — with the agentic-specific issues in a separate taxonomy, since OWASP now splits application-level and agentic risk into two documents.[20]

• Data poisoning and backdoors. An attacker who can influence training data — public web scrapers, RLHF preference data, fine-tuning sets — can implant a trigger: the model behaves normally until it sees a specific pattern, then executes attacker-chosen behaviour. This is the training-time analogue of a logic bomb, detectable only by auditing data provenance, because the resulting model passes ordinary evals.
• Prompt injection, direct and indirect. Direct: the attacker is the user, typing the malicious instruction themselves. Indirect: the attacker is a third party who plants the instruction somewhere the model will later read — a webpage, a document, an email, a tool’s return value — and the model executes it with the original user’s authority. Indirect injection is the more dangerous form because the user never sees the attack; the lethal trifecta almost always describes this variant.
• Jailbreaking. Distinct from injection: the attacker isn’t hijacking the model’s task, they’re persuading it to override its own training — role-play framing, encoding tricks, multi-turn erosion, appeals to fictional authority. It attacks alignment rather than task-following, which is why jailbreak resistance is a different engineering problem from injection resistance, even though both exploit the same instruction/data seam.
• Sensitive information disclosure and system prompt leakage. Models can regurgitate memorised training data verbatim under the right query, or be talked into revealing the system prompt, which often contains business logic, tool definitions, or credentials that shouldn’t have been there in the first place. The fix is defence in depth: don’t put secrets in the prompt, and assume the prompt is public.
• Model extraction. An attacker with only API access can query a model systematically and train a substitute that approximates it — stealing months of training investment through the interface you exposed for legitimate use. Rate limiting and output perturbation slow this; nothing stops it outright against a determined, well-resourced attacker.
• Vector and embedding weaknesses. RAG-specific: a poisoned document in the retrieval corpus doesn’t need to fool the model directly, it just needs to rank highly enough to get retrieved, turning your knowledge base into an injection vector. Embedding inversion is the mirror risk — reconstructing sensitive source text from stored vectors that were assumed safe because they “weren’t the original data.”
• Excessive agency. OWASP’s term for a harness that grants a model more permission, autonomy or tool access than the task requires. It’s the root cause underneath most severe agentic incidents — not a flaw in the model’s judgement, but a design choice about how much blast radius that judgement was given.
• Improper output handling. The output-side twin of injection: treating model-generated text as safe to execute, render or query without validation. Model output piped straight into eval(), a shell command or an HTML template reproduces every classical injection class — SQLi, XSS, command injection — with the LLM standing in for the untrusted user.
• Memory poisoning. For agents with persistent memory or logs that feed future runs, an attacker who can influence one session can implant false “experience” that corrupts every session after it. Any compounding knowledge store an agent writes to needs validation gates on what goes in.
• Confused deputy via tool calls. The agent uses its own legitimate credentials to do something the credential-holder never authorised, because the instruction came from untrusted content rather than the operator. Indistinguishable from normal operation in an access log, because technically it is normal operation.
• Unbounded consumption. Denial of service for the token-metered era: an attacker who can trigger expensive loops, huge context windows or repeated tool calls can run up compute cost or exhaust rate limits without touching sensitive data — “denial of wallet” alongside classical denial of service.
• Judge and verifier injection. If your evaluation or grading step is itself an LLM call, it inherits every vulnerability above. An attacker who can influence what the judge reads can manipulate the score it assigns, which matters enormously if that score gates a deploy or an autonomous action.
• Supply chain. Pre-trained weights from a public hub, a skill file, an MCP server, a framework dependency — all carry the same risk profile as any third-party package, except that review tooling for “is this checkpoint backdoored” is far less mature than review tooling for “is this npm package malicious.”
• Infrastructure. Sandbox escape, secrets, isolation, logging — the classical layer, unchanged in kind but raised in stakes, because the thing running inside the sandbox now has judgement and can actively look for the door left open, which is exactly what happened in July.

Supply chain deserves its own weight, because it’s where AI security stops being about prompts and starts being about ordinary dependency hygiene applied to a new kind of dependency. In March 2026 a backdoored version of LiteLLM — the gateway underneath CrewAI, DSPy and dozens of other agent frameworks — sat on PyPI for three hours. The attackers, a group publicly tracked as TeamPCP, had compromised a GitHub Action to steal LiteLLM’s own publishing token and push the malicious release themselves.[26] Skills, MCP servers and fine-tuned checkpoints are all dependencies in exactly this sense, and the discipline that applies to them is the discipline that already applies to any third-party package: pin, verify, review — the SBOM habit from classical practice, extended to artefacts that didn’t exist two years ago.

AI-enabled security: the model as actor

This is where the field stops resembling anything in the classical taxonomy — the model itself as the actor, not just the target. The clearest case study is July 2026. Agents running two OpenAI models were being evaluated on cyber capability, with safety classifiers deliberately lowered for the test. They used their only permitted network egress — a registry proxy — to build a message board between agent instances, and once one agent found a zero-day in the proxy, it shared the exploit for the others to reuse. That got them to the open internet, where they found a default API key on an unrelated benchmark app and used that foothold to breach Hugging Face’s production infrastructure, staying inside for roughly two and a half days — reading only cybersecurity datasets, which is what eventually gave them away to Hugging Face’s own AI monitoring.[27]

It wasn’t a one-off. In September, Google disclosed that Gemini had gained unauthorised access to three outside systems during a test, believing they were part of the test when they were actually connected to the internet.[28] Two labs, two months, the same failure: an agent treating a boundary as a puzzle to route around.

The design principle that follows: treat the model as a permanently untrusted component, and put every enforceable guarantee in the code around it, not in the model’s judgement.

• Loop. Every agent loop needs a hard ceiling — max steps, max wall-clock time, max spend — enforced outside the model’s control, not as a suggestion in the prompt. This is the direct fix for unbounded consumption, and the cheapest control on the whole list.
• Tools. Fewer tools is a security property, not just a UX one — Vercel found that cutting its agent’s toolset by 80% made it score higher, not lower.[32] Each tool should have the narrowest scope that does its job: a “search the web” tool, not a general HTTP fetch; a “read this repo” tool, not filesystem access. Typed, schema-validated arguments close a whole class of confused-deputy attacks, because a well-typed schema can’t smuggle an unrelated instruction the way a free-text argument can.
• Permissions. Provision every agent identity like a service account: a scoped credential, rotated regularly, with exactly the access its task requires and no more — the direct antidote to excessive agency. Consequential actions (sending money, deleting data, anything hard to undo) get a confirmation gate, either a human or a separate, more conservative verifier — never the same context that could have been injected.
• Observability. Log the full trajectory, not just the final output, and monitor it while the agent runs, not after. The July incident’s own root-cause analysis states plainly that trajectory monitoring wasn’t in place during the evaluation that went wrong. Detection should flag behavioural anomalies — an agent probing infrastructure it has never touched before — rather than relying only on content filters. That’s exactly the signal Hugging Face caught: the intruder was reading only cybersecurity datasets, which no legitimate process would.
• Context management. Tag every span of context with its provenance — user, system, retrieved document, tool output — and never let a downstream step treat a retrieved span as equivalent to a system span. Microsoft calls a lightweight version of this “spotlighting”: delimiting untrusted content clearly enough that both the model and any programmatic filter can tell the difference.[29] It’s not a guarantee, but it’s the highest-leverage cheap control available today.
• Verification. Prefer computational verifiers to inferential ones wherever the task allows — the same lesson as the eval harness in the previous part, now applied under adversarial pressure instead of just noisy conditions. A schema check, an allowlist match or a unit test can’t be talked out of its verdict the way an LLM judge can, which is exactly the property judge injection exploits.
• State and recovery. Stage consequential actions before committing — write to a draft, propose a diff, hold a transaction — so a compromised or hallucinated decision is reversible before it’s real. Recovery should default to rollback on anomaly rather than continuing forward: the inverse of what happened when the OpenAI agents treated a blocked path as a puzzle to route around instead of a boundary to respect.
• Orchestration. Where the task and threat model justify the cost, split planning from execution. The dual-LLM pattern, and Google DeepMind’s CaMeL[30] after it, are the strongest design available: one component plans from trusted input only; a separate one touches untrusted content and holds no tool access, and nothing crosses that seam without an explicit, checkable capability. For multi-agent systems the same principle rules out shared writable infrastructure between agents — the July message board only existed because the package-registry proxy was writable by every agent instance.
• Cross-cutting: supply chain. Everything reachable by the harness — weights, skills, MCP servers, framework dependencies — gets the discipline any dependency gets: pin versions, verify signatures where they exist, review before adopting, and keep the equivalent of an SBOM for what your agent can reach. LiteLLM is the concrete argument for why this isn’t optional anymore.
• Cross-cutting: adversarial testing before deploy. Red-team the harness, not only the model. In one Cursor CVE, the vulnerability wasn’t in the model at all — it was an allowlist that actively helped the attacker by auto-approving a poisoned command.[31] A secure development process for an agentic system needs a threat model that treats the harness’s own permissions, tools and control flow as attack surface, evaluated like any other piece of production infrastructure.

Everything above is the same move repeated at a different layer: don’t trust the model’s judgement where you can enforce a guarantee in code instead. That is also, not coincidentally, the discipline the next part needs — shipping reliably on top of a model that keeps changing underneath you.

6.CI/CD and continuous innovation

The model keeps changing underneath you. Shipping reliably on top of a moving target — and turning each model release into an upgrade rather than a regression — is the last piece of the job.

Classical CI/CD rests on one assumption: the only thing that changes between two releases is your code. Pin the dependencies, run the tests, and if they pass, the system behaves the way it did yesterday plus your change. AI breaks that assumption three ways. Your code changes. Your context changes — prompts, tools, skills, the retrieval index. And the model changes, sometimes on a schedule you chose, sometimes on one the provider decided.

Like security, this part splits into two halves that behave differently.

• Delivering AI — shipping a system with a model inside it: how CI/CD, automated testing, observability and monitoring change when three things move independently and you control only two of them.
• AI in delivery — the model inside your pipeline: agents that write, review, test, secure and operate ordinary software, and what they break along the way.

Delivering AI: CI/CD when the system has a model inside

The field got here in four eras.

• 1999–2010. Integrate continuously.
  Kent Beck put continuous integration into Extreme Programming in 1999, and Martin Fowler’s 2000 essay made it standard practice: merge small changes often, and let an automated build tell you within minutes if you broke something. Flickr’s 2009 “10+ deploys per day” talk and Patrick Debois’ first DevOpsDays turned it into a culture, and Humble and Farley’s Continuous Delivery (2010) turned it into a pipeline: every commit is a release candidate.[33] The assumption underneath was deterministic code — a test either passes or it doesn’t.
• 2015–2020. MLOps. The model is an artefact.
  Google’s “Hidden Technical Debt in Machine Learning Systems” (Sculley et al., 2015)[34] drew the figure every ML engineer knows: the model code is a small box in the middle of a much larger system of data collection, configuration, serving and monitoring. Google’s ML Test Score (2017)[35] turned it into a checklist, MLflow and Kubeflow (2018)[36] into tooling, and Accelerate (2018)[37] gave the field the four DORA metrics. Pipelines now versioned data and models as well as code — but the model was yours. You trained it, and you decided when it changed.
• 2023–2024. LLMOps. The model is rented.
  The model became someone else’s API. In July 2023 a Stanford and Berkeley study showed GPT-4’s accuracy at identifying prime numbers dropping from 84% to 51% between March and June — under the same model name.[38] It was the first clean demonstration that a pinned name is not a pinned behaviour. OpenAI Evals, LangSmith, Langfuse and promptfoo appeared around the same time:[39] prompts and evals started getting versioned like code, tracing was borrowed from distributed systems, and failures got a taxonomy to tackle.
• 2025–2026. AgentOps. The system writes code and acts.
  Agents moved into the pipeline itself — Claude Code and Codex running in GitHub Actions, opening and reviewing pull requests. The model is now both the thing being shipped and a contributor to it. In September 2025 Anthropic published a post-mortem of three infrastructure bugs that had quietly degraded Claude’s output for weeks — provider-side regressions that no model-version pin could have caught.[40] DORA’s 2024 report found that higher AI adoption correlated with lower delivery stability.[41] OpenTelemetry started standardising traces for generative AI,[42] and model releases went from quarterly to near-weekly.

What changes from classical CI/CD

CI/CD was built to answer one question: did my change break anything? AI engineering adds a second one every week: did someone else’s change break anything — and can I use it?

What changes, concretely:

• Everything that shapes behaviour is code. Prompts, tool schemas, skills, instruction files, judge prompts, retrieval config, the model snapshot. If it isn’t versioned, you can’t diff it, review it or roll it back. A release is not a commit hash; it’s a tuple — code, context, model, index.
• Tests are statistical, not binary. A test that passes nine times out of ten is a distribution, not a green checkmark. Gates run on pass^k and confidence intervals against a baseline — the same maths as the harness’s verification layer, now wired into the pipeline.
• Tests cost money. A full eval run with an LLM judge can cost more than the change it’s testing. So the suite is tiered, and only the cheapest tier runs on every commit.
• The dependency moves on its own. Pin model snapshots, never aliases. Track the provider’s deprecation schedule like any other end-of-life dependency. And run the suite when the provider changes, not only when you do.
• Rollback has more than one axis. A regression can come from the code, the prompt, the model or the index. Roll back one axis at a time, or you’ll never learn which one broke it.
• The monitor is a model too. Online evals use LLM judges, which inherit every weakness from the evals part and every attack from the security part. The grader needs its own evals.

The pipeline, in build order

The pipeline, in build order:

1. Version everything. Model pinned by snapshot, prompts, tools, skills, eval sets, judge prompts, index version — all in git, all reviewed like code.
2. Run the test pyramid.
3. Gate. Block the release on regression against the last release, per slice, plus cost and latency budgets. The gate compares to a baseline, not to an absolute number — “82%” means nothing on its own; “down 4 points on refunds” does.
4. Stage the release. Shadow first: the new version runs on real traffic, its output never shown. Then a canary to a few percent of users, with feature flags so a model or prompt can be switched without a deploy, and a rollback measured in minutes.
5. Observe and monitor.
6. Feed back. Low-scoring and user-flagged production traces become new eval cases. The eval set grows out of the failures you actually have, not the ones you imagined.

Two kinds of tests

An AI system has two kinds of tests, and most teams blur them. Deterministic tests cover everything around the model — harness code, tool implementations, output parsers, permission rules, schemas. These are written and run the classical way: unit, integration, property-based tests on tool arguments, and they must be green every time. They should be the majority of the suite. Record and replay model responses so harness tests don’t call a live model — they become fast, free and deterministic. Eval tests cover the model’s behaviour: pass rates on golden sets, scored against rubrics or by a judge. These are statistical by nature, and they run on a schedule — the pyramid from the evals part, now with a cadence attached:

• Unit, on every commit. Deterministic checks on single calls: schema valid, tool arguments well-formed, format correct.
• Integration, on every pull request. Single-agent task evals with a sampled LLM judge.
• End-to-end, nightly and on every model change. Full trajectories, multi-agent runs, and the red-team suite from the security part.

Two rules change from classical testing.

First, flakiness. The classical rule is that a flaky test is a bug in the test — quarantine it and fix it. In AI, some flakiness is the product. So separate the two: a flaky deterministic test is a bug; a flaky eval is a finding, and you measure it with k trials instead of hiding it. Don’t lean on temperature 0 or a seed to make it go away — providers don’t guarantee deterministic output even then.

Second, retirement. Golden-set cases are never silently deleted, only retired with a written reason, because a deleted case is a regression you’ve agreed not to see.

Monitoring and observability

Monitoring answers the questions you knew to ask. Observability answers the ones you didn’t. For an AI system, that means one trace per task, with a span for every model call, tool call, retrieval and sub-agent. Each span records the model snapshot and prompt version, the full input and output, tokens, latency and cost, tool arguments and results, and eval scores attached afterwards. In classical software a stack trace tells you where it broke. In AI the trace is the only record of why.

Use the OpenTelemetry conventions for generative AI[42] so your traces aren’t locked to one vendor — Langfuse, LangSmith, Phoenix and Braintrust can all read them. The same traces serve three jobs in this essay: debugging, the trajectory monitoring the security part asked for, and the feedback step that grows your eval set. Traces contain user data, so retention, access control and PII redaction are part of the design, not an afterthought. In the EU, for high-risk systems, logging isn’t only good practice — Article 12 of the AI Act makes automatic record-keeping a legal requirement.[43]

Monitoring itself runs on two layers of metrics, and you need both.

• Operational. p50 and p95 latency, error rate, cost per task, tokens per request, tool-failure rate, rate-limit hits.
• Quality. Online judge scores on sampled traffic, refusal rate, task completion, escalations to a human, user feedback.

Three kinds of drift are worth watching separately.

• Input drift. Users start asking for things your evals never covered.
• Output drift. Same inputs, different behaviour, because your context or your model changed.
• Provider drift. The model changed with no version bump — GPT-4 in 2023, Claude in 2025.

Set a quality SLO next to the latency one — for example, 95% of sampled answers score at least 4 on the rubric over seven days — and alert on drops per slice, not only on the average. And treat a cost spike as a security signal as well as a finance one: it is what the denial-of-wallet attack from the security part looks like on a dashboard.

AI in delivery: the model inside your pipeline

The other half runs the other way. Here the model isn’t the thing you ship — it’s one of the workers shipping it, in any software, AI or not.

• Write. Agents now author whole pull requests, not just autocomplete lines. The volume of code goes up, and the bottleneck moves from writing to review.
• Review. AI review is a good first pass: it catches obvious bugs, missing tests and style drift before a human looks. It can’t be the last pass. A model reviewing a model’s code shares its blind spots, and a reviewer that reads PR text can be injected through it.
• Test. Agents generate tests, diagnose failing CI runs from the logs and propose fixes, and triage flaky tests. The trap is tests that assert whatever the code currently does rather than what it should do — coverage goes up, confidence doesn’t. Mutation testing is the cheap check: if you break the code and the generated tests still pass, they were testing nothing.
• Secure. This is the defensive side of AI-enabled security from the previous part. In November 2024 Google’s Big Sleep found an exploitable memory-safety bug in SQLite before release — the first public case of an AI agent finding a previously unknown vulnerability in widely used software.[44] By 2026 Mythos was finding zero-days at machine speed. The same capability is available to attackers, so the scan has to run on every change, not once per quarter.
• Release and operate. Release notes, change logs, dependency upgrades, large migrations, on-call triage and incident summaries — the tedious parts of operations that agents are genuinely good at.

The costs are measurable. Google’s 2024 DORA report found that a 25% increase in AI adoption came with faster code reviews and better documentation, but also a 7.2% drop in delivery stability.[41] The researchers’ explanation is that AI lets teams ship more code faster, and larger changes are slower and more prone to instability. GitClear’s analysis of 211 million changed lines found an eightfold increase in duplicated code blocks during 2024, and for the first time, more code was copy-pasted than moved.[45]

And the pipeline itself became an attack surface. In December 2025 Aikido Security disclosed PromptPwnd:[46] GitHub Actions and GitLab pipelines running Gemini CLI, Claude Code and Codex could be hijacked, because the workflows passed untrusted text from GitHub issues straight into the agent’s prompt. The attacker needs no interaction — the workflow triggers itself on a new issue or pull request. This is the lethal trifecta from the security part, sitting inside CI: repository secrets as the private data, issue text from any GitHub user as the untrusted input, and write access to issues and PRs as the way out. The fix is the same as in that part — remove a leg. Privileged agents don’t run on untrusted triggers, tokens are read-only by default, and an agent that reads the public internet never holds write tools.

AI doesn’t retire the old rules of CI. It makes them matter more: small batches, fast feedback, tests you trust, and a human who owns the merge.

Clean code and technical documentation: code is context

A growing share of your codebase is now read by a model before a human reads it. Clean code used to be a courtesy to the next engineer; now it’s also context management for the next agent. Clear names, small modules, types and tests next to the code help an agent find the right file and make the right change. Messy code costs tokens, and a model that reads two contradictory patterns will copy the wrong one half the time. The duplication numbers above are what happens when nobody holds that line. Keep diffs small enough that review is real, and treat the test suite as the spec — it is literally the verifier from the scaffolding and harness part.

Documentation now has two audiences. For humans, keep decision records: why this model, why this threshold, why this piece of scaffolding. For agents, write instruction files — AGENTS.md, skills, llms.txt. The rule is the same for both: if a decision lives only in someone’s head, neither the next hire nor the next agent can respect it. And document every scaffold with the model weakness it patches — that note is the deletion ticket for the next release.

Continuous innovation: the model release ritual

Every model release, in order:

1. Diff the model. Run the full suite against the new model with your current harness. Where did it improve, where did it regress — per slice, with cost and latency next to the scores.
2. Try to delete scaffolding. Remove each patch whose weakness the new model may have absorbed, and re-run. If the score holds, the deletion stays. This is the moment the scaffolding’s expiry dates get checked.
3. Re-route. Update the router with the new model’s cost, performance and latency points, as in the auto-routing idea from the evals part.
4. Revisit the hiring decision. Run the first judgement call again: tasks that weren’t worth hiring the brain for may now be, and tasks you ran on the frontier model may now fit a smaller, cheaper one.
5. Ship it through the same pipeline. Shadow, canary, rollback ready. A better model is still a change.

At weekly release cadence you can’t do this manually. Automate the diff and the deletion attempts, and keep human judgement for the decision.

This is the whole job, reduced to one line each. Hire the brain only where it pays. Measure it rigidly. Build the least harness that holds it. Bound what it can do. Ship it on a pipeline that assumes the ground will move. The model will keep getting better on its own. Everything else in this essay is the part that won’t — and that is the job of AI engineering.

References

1. Shunyu Yao et al. ReAct: Synergizing Reasoning and Acting in Language Models (Oct 2022) — the reason → act → observe loop, published as a prompting method a month before ChatGPT’s launch. arXiv:2210.03629
2. Toran Bruce Richards, Auto-GPT (March 2023) — github.com/Significant-Gravitas/AutoGPT. Yohei Nakajima, BabyAGI (March 28, 2023) — github.com/yoheinakajima/babyagi.
3. Patrick Lewis et al. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (2020) — the original RAG paper. arXiv:2005.11401
4. Timo Schick et al. Toolformer: Language Models Can Teach Themselves to Use Tools (Feb 2023). arXiv:2302.04761
5. OpenAI. Function Calling and Other API Updates (June 13, 2023). openai.com
6. Answer.AI. Thoughts on a Month with Devin (Jan 8, 2025) — 3 of 20 real-world tasks completed end to end, a 15% success rate. answer.ai
7. Anthropic. Introducing the Model Context Protocol (Nov 26, 2024). anthropic.com/news/model-context-protocol
8. OpenAI. Introducing OpenAI o1-preview (Sept 12, 2024) — the first reasoning model. openai.com
9. Anthropic. Claude 3.7 Sonnet and Claude Code (Feb 24, 2025) — Claude Code’s research-preview launch, with terminal and permission-rule access. anthropic.com/news/claude-3-7-sonnet
10. OpenAI. Introducing Codex (May 16, 2025) — codex-1, a version of o3 tuned with reinforcement learning on real software-engineering tasks. openai.com
11. OpenAI. Building More with GPT-5.1-Codex-Max (Nov 19, 2025) — compaction across context windows for long-horizon coding tasks. openai.com
12. Yilun Yao, Xinyu Tan, Chao-Hsuan Liu et al. Harness-Bench: Measuring Harness Effects across Models in Realistic Agent Workflows (May 2026) — 106 tasks, 6 harnesses, 8 model backends; the top harness (NanoBot, 76.2) and bottom (OpenClaw, 52.4) span a 23.8-point gap under the same task set and model pool. arXiv:2605.27922
13. OpenAI. How Enabling Two Settings Tripled Our ARC-AGI-3 Scores — GPT-5.6’s public-set score rose from 13.3% to 38.3% via retained reasoning and compaction alone. openai.com · arcprize.org
14. Thariq Shihipar (Anthropic). The New Rules of Context Engineering for Claude 5 Generation Models (July 24, 2026) — over 80% of Claude Code’s system prompt removed with no measurable loss on coding evals. claude.com/blog
15. Arena.ai. Kimi K3 takes #1 in the Frontend Code Arena, ahead of Claude Fable 5 and GPT-5.6 Sol (July 2026). thenewstack.io · arena.ai
16. John Graham-Cumming, How to Beat an Adaptive Spam Filter (MIT Spam Conference, Jan 2004) — blog.jgc.org. Nilesh Dalvi et al., Adversarial Classification(KDD 2004). Daniel Lowd & Christopher Meek, Adversarial Learning (KDD 2005) — PDF. Marco Barreno et al., Can Machine Learning Be Secure? (AsiaCCS 2006) — PDF
17. Battista Biggio et al. Evasion Attacks against Machine Learning at Test Time (ECML PKDD 2013). arXiv:1708.06131 · Christian Szegedy et al. Intriguing Properties of Neural Networks (2013). arXiv:1312.6199 · Ian Goodfellow et al. Explaining and Harnessing Adversarial Examples (2014). arXiv:1412.6572
18. TechCrunch. Microsoft Silences Its New A.I. Bot Tay After Twitter Users Teach It Racism (March 24, 2016). techcrunch.com
19. Riley Goodside, original prompt-injection demonstration (Sept 11, 2022) — X/Twitter. Simon Willison, Prompt Injection Attacks Against GPT-3 (Sept 12, 2022) — simonwillison.net. CNBC, on the DAN jailbreak (Feb 2023) — cnbc.com
20. OWASP GenAI Security Project. Top 10 for LLM & GenAI Applications — genai.owasp.org/llm-top-10. OWASP Top 10 for Agentic Applications (2026) — genai.owasp.org
21. Simon Willison. The Lethal Trifecta for AI Agents (June 16, 2025). simonwillison.net
22. MITRE ATLAS — adversarial threat matrix for AI systems. atlas.mitre.org
23. Riyaz Walikar. An SSRF, Privileged AWS Keys and the Capital One Breach. blog.appsecco.com
24. 2017 Equifax Data Breach (CVE-2017-5638). en.wikipedia.org
25. Milad Nasr, Nicholas Carlini et al. The Attacker Moves Second: Stronger Adaptive Attacks Bypass Defenses Against LLM Jailbreaks and Prompt Injections (Oct 2025). arXiv:2510.09023
26. LiteLLM. Security Update: Suspected Supply Chain Incident (March 2026). docs.litellm.ai · Datadog Security Labs corroboration — securitylabs.datadoghq.com
27. Hugging Face. Agent Intrusion: A Technical Timeline (July 2026) — the OpenAI-agent evaluation that escalated into a breach of Hugging Face’s production infrastructure. huggingface.co
28. NBC News. Google Says Its AI Model Gained Unauthorized Access to Three Outside Systems (Sept 2026). nbcnews.com · securityweek.com
29. Keegan Hines et al. (Microsoft Research). Defending Against Indirect Prompt Injection Attacks With Spotlighting (2024). arXiv:2403.14720
30. Edoardo Debenedetti et al. (Google DeepMind). Defeating Prompt Injections by Design (CaMeL, March 2025). arXiv:2503.18813
31. GitHub Security Advisory. Cursor — Terminal Tool Allowlist Bypass via Environment Variables (CVE-2026-22708). github.com/cursor/cursor/security/advisories
32. Vercel. We Removed 80% of Our Agent’s Tools. vercel.com/blog
33. Martin Fowler. Continuous Integration (2000, rev. 2024) — traces the practice to Kent Beck’s Extreme Programming (1999). martinfowler.com. John Allspaw & Paul Hammond, 10+ Deploys Per Day: Dev and Ops Cooperation at Flickr (Velocity, 2009) — video. DevOpsDays, founded by Patrick Debois, first event Ghent, 2009 — devopsdays.org
34. D. Sculley et al. Hidden Technical Debt in Machine Learning Systems (NeurIPS, 2015). papers.neurips.cc
35. Eric Breck et al. The ML Test Score: A Rubric for ML Production Readiness and Technical Debt Reduction (2017). research.google
36. Databricks. Introducing MLflow (June 2018) — databricks.com. Kubeflow 0.1 announcement (May 2018) — kubernetes.io
37. Nicole Forsgren, Jez Humble & Gene Kim. Accelerate: The Science of Lean Software and DevOps (2018) — origin of the four DORA metrics. itrevolution.com
38. Lingjiao Chen, Matei Zaharia & James Zou. How Is ChatGPT’s Behavior Changing over Time? — the 84%→51% prime-number-identification drift, March–June 2023. arXiv:2307.09009
39. OpenAI Evals — github.com/openai/evals. LangSmith — langchain.com/langsmith. Langfuse — langfuse.com. promptfoo — github.com/promptfoo/promptfoo
40. Anthropic. A Postmortem of Three Recent Issues (Sept 17, 2025). anthropic.com/engineering
41. DORA. Accelerate State of DevOps Report 2024 — the 25%-adoption / 7.2%-stability-drop finding, cited twice in this section. dora.dev
42. OpenTelemetry. Semantic Conventions for Generative AI (2024–). opentelemetry.io
43. EU Artificial Intelligence Act. Article 12: Record-Keeping. artificialintelligenceact.eu
44. Google Project Zero. Big Sleep SQLite discovery (Nov 2024) — “the first public example of an AI agent finding a previously unknown exploitable memory-safety issue in widely used real-world software.” googleprojectzero.blogspot.com
45. GitClear. AI Copilot Code Quality 2025 — 211M changed lines analysed, an 8x rise in duplicated blocks, more code copy-pasted than moved for the first time in 2024. gitclear.com
46. Aikido Security. PromptPwnd: Prompt Injection Inside GitHub Actions (Dec 2025). aikido.dev