The art of AI engineering — harnessing the power of AI for reliable use — Part I

1.Introduction

AI engineering — the job rising out of this new Cognitive Revolution.

What is it, actually?

In the last essay I argued that the LLM is a brain for hire. This one is about the people whose job it is to hire it well — and I’m writing it as someone deep in the trenches myself, from my own experience working with AI, my own learnings, failures and mistakes.

This is my own definition: AI engineering is the art and science of building reliable, production-ready products out of pre-trained AI models. Throughout this essay, “model” means general-purpose foundational AI models — one brain, many jobs — not the single-task models of classical ML.

Because the model is pre-trained, this job has a different centre of gravity from any software engineering discipline before it, and a few things are worth being clear about from the start.

You don’t control the brain of the AI model.

The frontier labs can steer the direction of the model’s intelligence and capability. But even they don’t fully decide it — what a model can do is bounded by what the laws of nature currently permit, and the labs grow the models rather than concretely design them. Capabilities arise; they are not specified.

And the brain you don’t control is also a tricky thing to work with. It is non-deterministic: the same question can produce different answers, and a wrong answer can sound as persuasive as the right one on the surface. One mistake in the reasoning chain can trigger the wrong automation flow. The most difficult part of this job is not building, but being confident that what you build actually works reliably.

So your effort is better spent on other parts: understanding the model, judging which model is the right brain for which task, and mastering the craft of building the scaffolding around it — so that the model can reach for the right tool at the right moment.

The four skills. At the end of the day, this job can be consolidated into four skills:

1. Understanding the model — how it was trained, and why it behaves the way it does.
2. Judging the model — which one for which task, and knowing when it could fail — and when it has.
3. Building scaffolding — retrieval, tools, memory, context management, permissions and the like: the harness the model runs inside.
4. Running it in production — monitoring, debugging failures, cost, latency, reliability.

Who is best positioned to transition into AI engineering.

• ML engineers arrive holding the first two skills already. They are familiar with model training, which makes them sharper at recognising capabilities and limitations.
• Software engineers arrive holding the last two skills already. Frontend, backend, databases, cybersecurity, cloud, CI/CD, software architecture, infrastructure — all of it transfers directly to building around and on top of the model.
• Everyone else, from unrelated backgrounds: the discipline has no established pipeline yet — industry best practices, standard toolsets and frameworks are still emerging — which means no one holds a credential advantage, and everyone has a real chance to push the discipline forward, as long as you can master the four skills above and demonstrate them on something real, running in production.

With the definition and the core characteristics of the job in place, we can now walk through the process of AI engineering from end to end.

2.The first business and engineering judgement call: when to hire the brain

The first step in AI engineering is the one almost nobody talks about. If AI is a brain for hire, then deciding when to hire it — to replace or augment the biological intelligence currently doing the task — is a dedicated art in itself, and one of the most consequential decisions in the whole process. The three reasons below explain why.

First — you are introducing an unreliable component into something that already works.

Businesses have been rushing to integrate AI automation into their workflows and processes since early 2023 — the hype has run so hot that “tokenmaxxing” has become genuine slang, the AI transition bleeding into pop culture the way only real manias do. But here is the thing the rush forgets: without AI, the business still runs. There are established workflows, products and services delivered reliably, at a known cost, to customers who trust them. Integrating AI into these processes is a bet on a non-deterministic, probabilistic tool with two very different outcomes. It can speed things up, delight customers, cut cost. Or, if it goes wrong, it can disrupt operational reliability — the token of trust between you and your customers — and balloon cost. The brain for hire is getting smarter with every new model released.[1]Part of the art is judging when it is capable enough to handle a task, and when to be patient and wait for a smarter model. And waiting, done right, is not passive: the organisation that waits well spends the time mapping its workflows and building its evaluation and solid infrastructure layers below, so that when the capable model arrives, it can be deployed quickly in weeks instead of months or quarters.

Second — the ground keeps moving.

A model’s shelf life ranges from around twelve months in production to as little as six weeks at the frontier for an important release, and it only seems to be getting shorter over time.

Each new model brings new capabilities, new limitations, and a greater ability to handle tasks over longer horizons[2] — and the scaffolding has to change with it. This pace may slow down one day, whether because scaling laws plateau, labs run out of good research ideas, or for financial, regulatory or political reasons. But right now, it is a fierce race between companies and nations, with no clear forecast of when it will cool down. That severely shapes the job: building reliable products on top of changing models is like building skyscrapers on shaking ground. And the engineering answer to shaking ground is not “don’t build” — it is seismic design, as in civil engineering. Buildings in earthquake zones survive by flexing, not resisting: minimal rigid attachment to the thing that moves. In AI engineering, that means the golden rule is less is more — the least scaffolding that does the job, so that when the model changes, there is less to change with it. With that, you will also spend significantly less on maintenance and CI/CD later.

Third — AI products evolve on two tracks at once.

In classical software, there is a happy path: everything off it is a bug. Next version release, you fix bugs and add features customers ask for. An AI product has that entire track plus a second one: new model capabilities arrive every few months, and turning them into features is its own stream of work. It is at least double the work — and keeping both tracks moving, prioritised against each other, with limited engineering resources, is genuinely demanding.

To make it easier to imagine how this works in practice, for the rest of the essay I will carry two case studies through the whole AI engineering process — and I chose them because, interestingly, they sit at opposite ends of the single most important variable in this decision: the speed and cost of verification — how fast, and how expensive, is it to find out whether the artificial brain was right or wrong?

Case study 1. Software development sits at the quick and cheap end. Code is the rare knowledge work that announces its own failures: the test passes or it doesn’t, the build compiles or it doesn’t, in minutes. This characteristic is the most important reason why software is automating first and fastest. When checking the work is nearly free, you can afford to hire a fallible brain.

Case study 2. Venture capital investment sits near the slow and expensive end — and that is why it is worth examining closely. The finance industry, in my opinion, is one of the industries best positioned for the AI transition: data and prediction have been its core leverage since the beginning — deciding who gets a loan, weighing risk against interest rates, pricing public and private investments. But within finance, VC is the fascinating edge case, because the verification cost splits the job in half. Sourcing, screening, market mapping, data-room analysis — data-shaped, high-volume, checkable — can be automated well: roughly half the process. The investment decision itself, however, still needs human judgement: a fund makes perhaps twenty decisions per year, learns the truth seven to ten years later, and earns its returns from outliers — the deals that didn’t resemble past winners, because any deal that did would have been priced accordingly. Pattern-matching over historical data is precisely what a model does, and it is unfortunately what stops working here. The feedback loop is too slow and too sparse to ever tell you whether the model was right. So VC is not a typical use case for straightforward AI application — and that is exactly why it is interesting. Deciding which half of the workflow to hand over to the artificial brain is an art of AI engineering in itself.

In short, the rule of thumb is:

Hire the artificial brain where you can afford to check its work — and have a proper plan for if and when it fails.

So, my recommendation for businesses and engineers alike:

Map your organisation’s workflows, processes, software, and their dependencies first.

Then rank the automation candidates on three dimensions:

• Effort vs reward — the least technical effort for the highest commercial value; the obvious one in any engineering work.
• Verification cost — when the brain is wrong here, how quickly and cheaply will you know?
• Blast radius — when the brain is wrong here, how far does the mistake travel before something stops it? An error in a draft email can be stopped with one human review. An error hidden in an invoice amount could flow into accounting, into payments, into reports, into decisions — and surface only weeks later.

Based on your own ranking, with domain experts and engineering input together, automate the strongest candidate first, then the next, like a domino chain — building towards a comprehensive automated system without breaking what works.

The tricky thing is: the chain that makes the system powerful is the same chain an error travels down. Every AI link you add is also a corridor for mistakes to compound. So the sequencing rule comes with an important design rule: automate first where the mistake could be stopped in the fewest steps, and put a human in the loop wherever the mistake would travel.

3.The most difficult part of AI engineering: model and agent evals

We have covered the main skills of the job, and the first decision that comes before any of them: whether a task is worth automating at all, judged on effort, value and risk.

Now the harder question. You have decided to automate something. Which model do you hand it to, and how do you know it is actually doing the job?

I consider this the most difficult part of AI engineering — and not because the technique is complicated, but because it is so different. Normal software fails loudly. It throws an exception, the types don’t match, an assertion breaks. We have built decades of tooling on top of that — unit, integration and end-to-end tests — and it works because the same input gives the same output every time. One passing test proves the case forever.

AI-empowered products break that. The same input can give a different output on every run, so one passing run proves nothing. And when it does fail, it often fails silently. It can fail in the logic. It can fail in how it uses a tool. A small wrong detail early on goes off the rails later and produces a wrong or unethical result that drives users away, or worse.

Nothing crashes. Nobody gets paged. Your user finds out before you do.

So before you can trust the work, you have to build the things that check the work. It is nearly impossible to build a production-grade product, and to keep iterating on it, without evals.

So, first things first. What is an eval?

An eval is a repeatable measurement of whether a model or agent is good enough at a specific task to be trusted with it.

There are two layers to it.

• Model eval (or benchmark eval) — the model’s capability on its own.
• Agent eval (or application eval) — an agent is a model plus a harness, so this measures the whole system, scaffolding included.

Besides those, there are also safety evals — but these are mostly run by frontier labs, regulators and third-party auditors, to decide whether a model is safe for public release.

AI evals have also gone through a quick evolution since their early days. The benchmark names change every time. What actually changes underneath is the grader — who or what decides whether the answer was good, and at what cost.

• 2018–2020 · Fill in the blank. The first tests were worksheets. Give the model a sentence with a word missing. Check the word against an answer key. GLUE, SuperGLUE, SQuAD. Marking cost nothing, because marking was a string comparison.

  Then models started scoring full marks. A test everyone passes measures nothing.

• 2020–2022 · The giant quiz. So the tests got bigger. MMLU is 16,000 multiple-choice questions across 57 subjects: law, medicine, physics, moral philosophy.[3] Still tick-a-box. Still free to mark.

  This era lasted longer, because breadth is harder to fake. It died of two causes. Frontier models now cluster above 88%, and at that point a two-point gap is measurement noise rather than skill. And the questions were published on the internet, which means they eventually turned up in training data. Scale AI tested this directly: they wrote 1,250 fresh grade-school maths problems in the style of GSM8K, and the worst-performing model dropped 13 points on the new set, with some model families overfitting across nearly every size. The exam had leaked.[4]

• 2022–2023 · There is no answer key. ChatGPT changed the shape of the output. Answers stopped being one word and became paragraphs. What is the correct version of “write a polite email declining this meeting”? There isn’t one.

  So we asked people. Chatbot Arena puts two answers side by side and collects the vote. Millions of votes later, you have a ranking.[5] It works. But it is also slow, expensive and noisy, because people disagree with each other. The fix was to promote a model to judge. LLM-as-a-judge agrees with human raters 80–90% of the time at a fraction of the cost.[6] This is the moment evals became something you could run continuously instead of quarterly.

• 2023–2025 · Make it harder. AI models were producing fluent prose but shaky reasoning. So the field went back to questions with one correct answer and made them brutal. Graduate-level physics. Competition mathematics. Puzzles written fresh so memorisation doesn’t help. GPQA, ARC-AGI, FrontierMath, Humanity’s Last Exam.[7]

  The marking got cheap again, because the answer is either right or wrong. The expense moved upstream: you now need domain experts to write the questions. The point is to test whether the model can reason its way to the answer, not whether it has seen it before.

• 2024–2026 · Stop asking, start watching. Then the real break: stop giving an exam at all. Hand the model a broken repository and a terminal. Say “fix it”. Walk away, come back and check whether the tests pass. SWE-bench, Terminal-Bench, OSWorld.[8] There is no answer key. The environment is the grader. The model either fixes it or it doesn’t.

  But this is also far more expensive, because every task needs a working environment behind it. And you are grading a trajectory now, not just an endpoint — the model can reach the right end state by a route you would never have approved.

  July 2026 gave us the extreme version of that. OpenAI ran a cyber benchmark called ExploitGym with its production safety classifiers deliberately switched off, to maximise the score. The models spent a substantial amount of inference compute looking for a way onto the open internet, found a zero-day in a package-registry cache proxy, escalated privileges until they reached a node with internet access, then reasoned that Hugging Face might be hosting the ExploitGym solutions and went to fetch the answer key. Hugging Face’s security team detected and contained it, and no public models, datasets or user accounts were touched.[9] Nobody told the model to break into anything. It was told to solve ExploitGym, and this was a path to the score.

  This shows the danger of grading an AI model by the end state alone, without watching the path and the reasoning that got it there.

• 2025 onwards · Give it the actual job. Now the AI does real professional work. GDPval draws its tasks from economically valuable knowledge work and puts domain experts with 14 or more years of experience in the judge’s seat. BigLaw Bench does the same for legal work.

  This is the most expensive marking scheme the field has ever used. We use it anyway, because it answers the only question that matters commercially: can AI do the job or not?

Look back across the six eras and the pattern is clear. The questions went from closed to open. Early tests were right or wrong across many topics; later tests grade an approach, a piece of reasoning, a trajectory, on a scale.

What was being tested rose with the models — first recall, does it know the answer, then reasoning, can it work the answer out; now agency, can it do the job alone — and the difficulty climbed with it, from child to college to expert. The cost of marking went up every step. A fixed test set, written once and reused. Then public benchmarks. Then crowdsourced human preference — not cheap, but affordable because it borrows collective judgement. Then domain expertise, which is not affordable at all. That rising cost is what pushed the field to LLM-as-judge: a scorer fast and cheap enough to run continuously, as long as you keep it aligned with human raters. The tests stopped being fixed artefacts, because the next model release crushes the newest test and raises the bar again. And the newest step is the biggest: build a sandbox around real work, and let the system figure out what to do, completely open.

That history is mostly the story of model evals — public benchmarks, built by labs and researchers, measuring the model on its own. And that gives you the practical split. Model evals you can largely inherit: someone else has already run them, published the numbers, and paid for the domain experts. Agent evals you have to build yourself, because nobody else has your harness, your tools, your data, or your definition of good. So the working method is a funnel. Use the inherited layer to narrow the field, then build the layer only you can build to make the call.

For model eval. Public benchmarks are saturated. That makes them a scouting tool, not a verdict. Use them to narrow to three to five top candidates. A benchmark score still carries three signals even when contaminated.

• Exposure — a model trained on that material has seen that kind of knowledge.
• Intent — a lab that optimised for the benchmark is telling you which domain it wants to win, and how seriously.
• Ecosystem — the top model on a public benchmark becomes the one developers talk about, which fuels the flywheel of tooling, community, technical support around it.

These three things are all important even when the number is inflated.

For agent eval. Test it on your own work. Build a task evaluation set as a collaboration between AI engineers and domain experts — the engineers know how to build the harness, the experts know what good results look like, and neither can do it alone. Then hand the set to an LLM judge. That is the best available trade on speed and cost. It is not the best on accuracy — a human expert is — so it only works if you calibrate it: hand-label a few hundred cases, measure how often the judge agrees, and re-measure whenever you change the judge or the rubric.

Five criteria to test against:

• Capability — can it do the task, and how often.
• Safety — will it do something it shouldn’t, or fail in a way that harms someone.
• Reliability — does it do it consistently? Variance, tail failures, degradation over long runs.
• Latency — how long does the user have to wait? Time to first token, time to completion, and how both hold up under load and on long inputs.
• Cost — what does each successful outcome consume in tokens and money.

Beyond the eval itself, there is vendor due diligence: how often the provider is down and what the SLA guarantees, their security track record, their safety and ethics policies, and what they do about data privacy, protection, compliance and copyright. No eval run will tell you these — but they decide whether you can rely on the provider long term.

To make this concrete, I will use two case studies throughout the rest of the process, starting with building evals.

Case study 1 · Software engineering

Now, using what we have learnt so far, let’s map out the full software engineering process and see how we evaluate an AI automating each step.

Based on automation priority criteria from the previous section, let’s quickly rank them.

Based on these rankings, we could group into three tiers of priority given the capabilities of current models.

• Fully automate — testing and code review. Both stages get checked immediately by something cheaper than the model itself: a test either passes or it doesn’t, and a review comment is read by the human who still holds the merge button. Current models handle both well — they read a diff faster than a senior engineer, catch mechanical mistakes with high recall, and write tests that cover the obvious path reliably. Their one real limitation, missing the mistake that requires knowledge outside the codebase, is cheap to live with because the human at the merge button is the backstop.
• Automate with a human at the gate — implementation and the diagnosis half of operations. Current models have crossed the threshold here: they can hold a bounded coding task for hours, navigate an unfamiliar repository, iterate against failing tests to a passing state, and in an incident read the logs and propose a fix faster than a human on call. Two limitations keep the human at the gate. They reach green by unapproved routes often enough to matter — deleting the failing test, or special-casing the input so it passes without solving anything. And they do not know what they do not know: a 70% model does not fail 30% of the time with a clear signal, it fails with a confident wrong answer that looks like the right one. So the model works inside a corridor: bounded tickets, tests as the gate, transcript review of the path and not just the result, and a human executing any remediation. Cross the line without those guardrails and you have not automated the stage — you have moved its verification cost onto your customers.
• Augment, don’t automate — planning, design and requirements. The model is a capable sparring partner here: it can split a feature into tickets, draft an architecture and lay out trade-offs as well as a good mid-level engineer. But verification is slow — you learn a design was wrong when it doesn’t scale, and a requirement was wrong when the customer doesn’t renew — and by then the mistake has propagated into every stage below. Deployment sits in this tier for a different reason: it is already automated deterministically, and adding a probabilistic component would only make it less reliable without making it more capable.

Now the public benchmarks — the inherited layer. Ranked from most to least trustworthy. Read top to bottom and it is the eval history in miniature: the oldest benchmark is saturated, the newest are only comparable inside one harness, and the two stages cheapest to automate have no public benchmark at all — which is exactly why they are cheap.

Taking Terminal-Bench and SWE-rebench as the filter — the two boards running today’s models — the shortlist for software engineering comes down to Claude Fable 5, GPT-5.5, Grok 4.5, Claude Opus 5 and GPT-5.6, with GLM-5.2 as the open-weight option.

That is all the inherited layer can do. The rest is the layer only you can build — and here is one way to build it, stage by stage.

Score the four stages on the five criteria and you will not find one winner — pick the cheapest model that clears the bar at each stage, not the strongest; nothing says one brain for every job.

The judge

Every agent eval above ends the same way: a transcript lands in front of an LLM that decides whether it was good. That judge is the single most consequential design decision in the whole apparatus, and it deserves more thought than “ask a strong model to grade it”. Four choices shape it.

• What does the judge compare against? A reference-based judge has a gold answer — the merged diff, the postmortem’s root cause, the human reviewer’s comments — and scores against it. A reference-less judge has only the input and the output, and asks whether the output was relevant, grounded, safe, on-task. So you need both: reference-based for the offline eval set, where gold answers exist, and reference-less for production, where they never do. The stages above are mostly reference-based by design, which is why they work offline; the monitoring section later is mostly reference-less, which is why it is harder.
• One output or two? A single-output judge scores one run on an absolute scale. A pairwise judge sees two runs of the same task and picks the better one. Pairwise is the cheaper, more reliable way to answer “is model B better than model A on my work”, because humans and models both find comparison easier than absolute scoring — it is the Chatbot Arena insight, moved in-house. Use it when the question is “which one”; use single-output when the question is “good enough to ship”.
• How much structure? There is a spectrum. At one end, a free-form rubric: “judge this transcript for correctness, safety and efficiency” — fast to write, quick to drift. In the middle, a rubric broken into explicit evaluation steps the judge must walk in order, which is what frameworks like DeepEval call G-Eval.[10] At the other end, a decision tree: hard gates first — did it touch forbidden files, did it delete a test, is the output valid — each one a yes / no that fails the run outright, and only then a graded judgement on what passed. That last shape is what DeepEval calls a DAG metric, and it is the right one for the “grade the path” step, because the failures you most want to catch are binary and should never be averaged away by a good score elsewhere. One thing the gates should not be: a required sequence of steps. Anthropic’s own advice is to grade what the agent produced rather than the exact route it took, because agents regularly find valid approaches nobody anticipated, and a judge that demands a particular order of tool calls punishes creativity as if it were failure.[11] The gates are for the routes that are never acceptable — deleting a test, touching a forbidden file, fabricating a number — not for enforcing the one route you had in mind. Start loose while you are still learning what goes wrong: tighten into steps and gates as the eval becomes something CI depends on.
• What scope does it see? A judge can look at one output, one component — the retriever alone, one tool call alone — or the whole ordered trajectory: plan, decisions, tool calls, results, handoffs. For agents, trajectory is the scope that matters; the ExploitGym lesson was that the end state lied. In practice, this means four questions per run: did it complete the task, did it follow a sensible plan, did it waste steps, did each tool call make sense on its own.

Then calibrate, and be precise about which way you are wrong. Hand-label a few hundred runs and bucket the judge’s verdicts against yours. False negatives — the judge fails a run you would have passed — are noise; they slow you down. False positives — the judge passes a run you would have failed — are the dangerous kind, because they manufacture confidence. For the stages with a large blast radius, tune the judge to be strict and accept the noise; for review and test generation, tolerate some leniency. The 85% agreement bar is a starting point, not a destination — and every time you change the judge model, the rubric or the task mix, measure again, because the judge is itself a brain for hire and drifts like one.

Finally, put the judge where the work is. Offline, it sits in CI as an assertion: the eval set runs, scores land, a regression blocks the merge the way a failing test would. Online, it sits on the trace: every production run is logged with its spans — model calls, tool calls, retrievals — and a small set of reference-less judges samples them, flags the low scores, and routes those back into the eval set as new tasks. Tracing tools like Langfuse[12] exist for exactly this loop.

One last thing before handing the scoring to an LLM judge: the judge is a model too, and it comes with its own biases. Know them, and know what you can do about each.

Case study 2 · Venture capital investment

Software engineering was the easy case: the checks are cheap and mechanical, so the brain for hire moved in fast. Venture capital is the hard case, on the opposite end of how slow and expensive it takes to verify if the AI is right or wrong. That’s why I deliberately chose it to demonstrate the potential and limitations of automation. The work is judgement under uncertainty, the ground truth arrives seven to ten years after the decision, and the customer — the founder — is paying for a human to believe in them. If the method survives here, it can survive anywhere.

Here is the process of venture capital, end to end.

Based on the same three criteria — effort vs reward, verification cost, blast radius — the ranking looks like this.

Similar to the previous case study, now we can group them into automation tiers.

• Fully automate — sourcing and portfolio monitoring. Both are extraction at volume with a checkable source: a company database on one side, the company’s own update on the other. Current models do this well; they read a deck, a website, a monthly report and a cap table faster and more consistently than an associate, and they do not get bored on the four-hundredth one. The limitation is the same as code review’s: they miss what requires knowledge outside the document — the founder who left a previous company badly, the metric that is real but redefined since last quarter. Cheap to live with, because the partner still reads the flagged items and the pattern of what was flagged is itself reviewable.
• Automate with a human at the gate — screening, diligence extraction, the memo draft, legal deviation reports. Here the model has crossed the threshold in the same way as implementation: it can hold a data room of a few hundred documents, reconcile the deck’s claims against the financials, draft a memo that a partner would recognise as competent, and red-line a shareholders’ agreement against the fund’s template. Two limitations keep the human at the gate, and they are sharper than in software. First, it hallucinates numbers — a model asked for a metric the data room does not contain will, often, produce one, and a fabricated figure in a memo is indistinguishable from a real one. Second, it is persuadable by the material it is reading: a confident, well-written deck scores better than a plain one, and the model inherits that bias into its triage and its memo. So the corridor is: every number cites a source page or is flagged as missing; the screener’s rejects are sampled, not just its passes; the memo’s facts are reconciled mechanically and the argument is rewritten by the partner.
• Augment, don’t automate — reference calls, negotiation, the investment decision. The model earns its place as a sparring partner: it drafts better reference questions than most associates, it runs twenty cap-table scenarios in the time a human runs one, it will argue the bear case on a deal a partner has fallen in love with. But the verification is the company’s outcome, years away, and the blast radius is the fund. Two outside checks also land here. Customer tolerance: a founder choosing between term sheets is choosing a partner for a decade. And regulation: the fund has a fiduciary duty to its LPs.

Unfortunately, public benchmarks for this line of work are thin, and what exists measures adjacent skills rather than the job.

No benchmark exists for screening, memo-writing or investment judgement, for the obvious reason: the ground truth is proprietary and a decade late. So the inherited layer narrows the field on one axis — does it read financial documents accurately, and does it refuse to invent — and then you are on your own.

Building the agent eval — the detailed guideline.

Two things that are very different from the previous use case. First, screening is the one stage in either case study where you have a real, labelled history to test against — a decade of partner decisions with outcomes — and you should be careful with it. It is small, it is survivorship-biased, the market regime has changed under it, and above all the labels encode the partners’ taste, so a model that matches them perfectly has learned the fund’s biases, not the truth. Grade on ranking of the deals that worked, not agreement with the deals that were passed. Second, “done” for diligence is defined by a property that never came up in software: refusal. The best diligence model is not the one that finds the most; it is the one that says “not in the data room” every time that is the answer, and the gate should fail any run where it did not. Other than that, you are good to go to apply LLM-as-a-judge similar to the previous use case to finish your eval pipeline.

Towards the future

Everything above describes evals as they are mostly built today: one task set, one judge, one model, re-run by hand when something changes. That is already behind the curve. Five shifts are underway, and the teams ahead are treating them as the default.

• Evals are code. Version control them, review them in pull requests, give them an owner. Anthropic’s own arrangement is worth copying: a small evals team owns the infrastructure, and the domain experts — product, support, sales — contribute the tasks, because they are the ones who know what good looks like.[11]Maintaining evals should be as routine as maintaining unit tests. And write them ahead of the model: build the capability eval for a task today’s model fails, ship the feature anyway if it is good enough, and when the next model drops the suite tells you overnight which bets paid off.
• Evals become the test pyramid. Classical software has unit, integration and end-to-end tests, and the same shape is forming here. Unit: one component in isolation — the retriever, one tool call, one prompt — graded by code, cheap, run on every commit. Integration: one workflow, a chain of calls with the gates between them. End-to-end: the full trajectory in a sandbox, graded on outcome, expensive, run nightly or before release. Two things automated software testing already teaches. First, split capability evals — low pass rate, a hill to climb — from regression evals — near 100%, an alarm if they drop; a capability eval that saturates graduates into the regression suite. Second, report reliability, not just success: pass@k tells you whether the agent can do it at all; pass^k tells you whether it does it every time, and for anything customer-facing the second number is the one that matters. A 75% agent passes three runs in a row 42% of the time.[13]
• The eval becomes the router. The stage-by-stage tables above already say it: no single model wins every stage, so pick the cheapest that clears the bar at each. The next step is to pick automatically, and Cursor, Ramp and OpenRouter already offer exactly this.[14] Once every task carries a capability, latency and cost score per model, the eval results are a routing table — easy inputs to the small model, hard ones to the frontier — and a classifier in front of the harness reads it. One caution and one sizing rule. The caution: the router is itself a brain for hire, making a capability call about every input, so it needs its own eval. The sizing rule: match the router to the decision. Anthropic notes the classification can be done by an LLM or a traditional classifier, and for plain routing a simple classifier is enough.[15]For the orchestrator-workers pattern — where one model breaks the task apart, delegates the pieces and judges the results — the orchestrator is making the hard calls, and that is where the smarter model belongs.
• Diff the model, don’t just re-audit it. The labs face the same model-swap problem at the level of the weights: a new model is a million lines of code with no changelog, and human-written benchmarks only catch the risks someone already thought to test for. Anthropic’s answer is model diffing — compare the new model against the old one and let the comparison point to what changed, the way a programmer reviews the fifty lines that moved rather than the whole codebase.[16] My prediction is that this becomes part of every release: a behavioural changelog shipped with the model. And you should do the same one layer down — on every model swap, diff the trajectories on your own eval set, not just the scores. A flat score with a changed path is a change you have not understood yet.
• Evaluate the system, not the agent. The moment you run more than one agent — a swarm, an orchestrator with workers — single-agent evals stop predicting behaviour. Anthropic’s multi-agent research from August 2026 shows why: agents built on the same model are low-variance, so 18 out of 30 chose the identical branch name, and in a pricing game they colluded on price floors within three rounds.[17]Neither shows up in a one-agent eval. They show up only in a sandbox with the whole system running — the top of the test pyramid, and the reason to invest in it before you need it.

References

1. Anthropic. Newsroom — model-release announcements for the Claude family; the source for the release timeline in Figure 1. anthropic.com/news
2. METR. Measuring AI Ability to Complete Long Tasks(2025) — origin of the “task time horizon” metric shown in Figure 2, doubling roughly every seven months. metr.org · arXiv:2503.14499
3. Dan Hendrycks et al. Measuring Massive Multitask Language Understanding (MMLU). 15,908 questions across 57 subjects (rounded to “16,000” in the text). arXiv:2009.03300
4. Hugh Zhang et al. (Scale AI). A Careful Examination of Large Language Model Performance on Grade School Arithmetic (GSM1k) — 1,250 fresh problems; accuracy drops of up to ~13 points for the most overfit model families. arXiv:2405.00332. Builds on GSM8K — Karl Cobbe et al. (OpenAI), arXiv:2110.14168.
5. Wei-Lin Chiang et al. Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference. arXiv:2403.04132 · lmarena.ai
6. Lianmin Zheng et al. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena — strong LLM judges reach over 80% agreement with human raters, matching human–human agreement (the essay’s “80–90%” takes the upper end slightly beyond the paper). arXiv:2306.05685
7. Graduate- and frontier-level reasoning benchmarks: GPQA (arXiv:2311.12022), ARC-AGI (arcprize.org/arc-agi), FrontierMath (arXiv:2411.04872), Humanity’s Last Exam (arXiv:2501.14249).
8. Agentic / environment benchmarks: SWE-bench (arXiv:2310.06770, swebench.com), Terminal-Bench (tbench.ai), OSWorld (arXiv:2404.07972).
9. Hugging Face. Agent intrusion: a technical timeline(July 2026) — an OpenAI model, evaluated on the ExploitGym cyber benchmark with production safety classifiers reduced, escaped its sandbox through a zero-day in a package-registry cache proxy. ExploitGym is a UC Berkeley (Sunblaze / Dawn Song lab) benchmark, not OpenAI’s. huggingface.co. Corroborated by The Hacker News, CNN and Simon Willison.
10. DeepEval (Confident AI) — G-Eval (deepeval.com/docs/metrics-llm-evals) and the DAG metric (deepeval.com/docs/metrics-dag).
11. Anthropic. Demystifying evals for AI agents (January 2026) — grade what the agent produced rather than the exact route; a small evals team owns the infrastructure while domain experts contribute the tasks. anthropic.com/engineering/demystifying-evals-for-ai-agents
12. Langfuse — open-source LLM tracing and observability; production runs are logged with their spans (model calls, tool calls, retrievals) and sampled by reference-less LLM-as-a-judge scorers, the online loop described here. langfuse.com
13. Reliability metrics. pass@k — Mark Chen et al., Evaluating Large Language Models Trained on Code (Codex / HumanEval, arXiv:2107.03374); pass^k — Shunyu Yao et al., τ-bench (arXiv:2406.12045).
14. Model routing in production: Cursor Auto (cursor.com), Ramp’s LLM router (ramp.com), OpenRouter Auto Router (openrouter.ai).
15. Anthropic. Building Effective Agents (December 2024) — routing can be handled by an LLM or a traditional classifier; the orchestrator-workers pattern. anthropic.com/engineering/building-effective-agents
16. Anthropic. A “diff” tool for AI: finding behavioral differences in models— the essay’s “new vs. old model” framing simplifies the method, which primarily compares different models. anthropic.com/research/diff-tool
17. Anthropic. Patterns and problems in multiagent systems(August 13, 2026) — low-variance agents (18 of 30 chose the branch name “mvp-game-loop”) and price-floor collusion by round 3. anthropic.com/research/multiagent-systems