Can this proof of concept reach production?
Most agentic proofs of concept get judged after the demo goes well, which is the one moment nobody is able to judge them. Twelve questions, scored before you build. Each one says what earns a 0, a 1 and a 2, so two people scoring the same thing land in the same place.
12 questions · 4 sections · max 24 · one hard gate
The rules that decide whether this is useful or theatre.
- Score what is true today, not what is planned. "We'll build that in the pilot" is a 0. The screen measures readiness, not intent — this is the single rule that decides whether the exercise is useful or theatre.
- Everyone scores privately, then reveals. Read the question aloud, each person commits to a number, then compare. Discussing first collapses the room onto whatever the loudest person thinks.
- A two-point spread is the finding. Where scores disagree by two, the team does not share a picture of the system. Resolve that before moving on; it is worth more than the total.
- A blank is a 0. If nobody in the room can answer, nobody knows, and not knowing is the condition being measured.
- Re-score after narrowing, not after arguing. The legitimate way to move a score is to change the scope of the POC.
The work
Whether the task is shaped for an agent at all. These three fail quietly — nobody notices until the evaluation has nothing to measure against.
Repetition is what amortises the build. High-variance, low-volume work never earns back the evaluation and review cost.
If the list is open-ended, so is the blast radius, and no reviewer can approve it. Count the actions before you count the features.
Closed tickets, ledger entries, prior decisions. No recorded ground truth means no evaluation, and no evaluation means the demo stays the only evidence you will ever have.
The blast radius
Hard gateWhether you survive being wrong. A zero on any row here is a hard stop regardless of the total — this is the section that ends careers, not roadmaps.
Refunds, sends, deletes, commitments. The classic failure is a tool call that times out rather than fails, leaving the system unsure whether the action happened — and a sensible retry that does it twice.
Silent wrongness is the expensive kind. If nothing downstream catches it, the evaluation is your only defence and it will not be enough in week one.
A kill switch nobody owns is not a kill switch. A trace you cannot replay is not an audit trail.
The evidence
Whether you will be able to prove it worked. Decided before the build or not at all — you cannot retrofit a baseline.
Without a measured baseline, “better” is an opinion, and the pilot review becomes an argument about anecdotes.
These are the cases a demo never contains and production contains hourly. If you cannot assemble them now, you will not assemble them under launch pressure.
Timeouts, partial writes, ambiguous responses. Most agent incidents live here, and almost no POC tests it.
The ownership and the economics
Whether it survives contact with the organisation. Pilot purgatory is an ownership problem far more often than a tooling one.
If no one's metric moves, no one defends the budget at the next planning round, and the pilot quietly ages out.
If a data programme has to land first, that programme is the real project and it is six months long. Say so out loud now.
At a hundred times the volume the model is usually the smallest line. Review labour is usually the largest, and it is the one that scales linearly.
Agree these before you score, not after.
A total that lands one point above a line is a narrow, not a yes.
Four ways to answer most of these without the agent existing.
In ascending order of cost. Every one of them is cheaper than a pilot that fails in front of a customer.
Write the evaluation set before the spec
Ask for fifty real cases from last quarter with known-correct outcomes. Recorded ones, not invented ones. If the team cannot assemble fifty in a week, that isn't a scheduling problem: the task has no ground truth, and an agent you cannot grade is an agent you cannot ship. The failure to produce the set is the verdict.
Put a human behind the curtain
Before building anything, route real requests to a person sitting behind the interface the agent would use, answering under the same rules and with the same tools. You learn the true distribution of requests, which ones have no clean answer, and whether anyone uses the thing at all. You end the fortnight holding a labelled evaluation set you didn't have to imagine. Check the disclosure question with legal first if the flow is customer-facing.
Break the tools on purpose
In staging, make the tool time out rather than fail. Make it return success for an action that never happened. Make it return the wrong thing, confidently. Chaos engineering has been standard for services for twenty years and is almost unheard of for agents — which is odd, because this is precisely where agent incidents live.
Run it in shadow
Put it on live traffic and forbid it from acting. Log what it would have done, compare against what the human actually did, and review only the disagreements — usually a small fraction of cases, and all of the information. A real accuracy number with a blast radius of zero.
The screen is the easy half. The argument is the other one.
A blank scorecard is genuinely useful and costs nothing to give away. What it cannot do on its own is settle the rows where two people scored two points apart, which is where the real disagreement about the system lives. That part takes a room.
See the cohort → All resources →
Published 13 September 2026 · free to use and to pass on