The POC Selection Tool
Score an agent proof of concept before you build it, and find out whether it can reach production. Twelve questions in four short steps. Each one says what earns a 0, a 1 and a 2, so two people scoring the same idea land on the same number.
12 questions · 4 sections · maximum 24 · one hard gate
Built by Sunil Mathew, co-authored with Claude
From The Living Craft, practical learning and consulting with Sunil Mathew. Explore the programmes
- You have an idea for an AI agent. Before anyone builds it, this tool tells you whether it can reach production.
- Twelve questions in four sections. Each answer scores 0, 1 or 2. The maximum is 24.
- Section B is a hard gate. A 0 on any of its three questions stops the proof of concept, whatever the total.
- The result is one of three outcomes: pilot candidate, narrow it and score again, or a demo rather than a pilot.
- Pick one proof of concept. Score one at a time.
- Work through the four sections with Next and Back. Each has three questions.
- For each question, choose the answer that is true today, not the one that is planned. The page moves to the next question for you.
- Watch the score bar and the progress bar. Both stay at the top and update as you answer.
- Read the result at the end. It gives the outcome and what to fix first.
- Download the tool as an Excel workbook to score the next one offline. The PDF of your scored copy is at the end.
Five rules that decide whether this is useful or theatre.
- Score what is true today. "We will build that in the pilot" scores 0. The tool measures readiness, not intent. This one rule decides whether the exercise is useful or theatre.
- Score alone, then compare. Read the question aloud. Each person picks a number before anyone speaks. Talking first pulls the room toward the loudest voice.
- A two-point gap is the finding. If one person scores a question 0 and another scores it 2, the team does not agree on what the system is. Settle that before moving on. It is worth more than the total.
- A blank is a 0. If nobody in the room can answer, nobody knows. Not knowing is what the question measures.
- Change the scope, not the argument. The right way to raise a score is to narrow what the proof of concept does, then score it again.
Task fit
Is this task shaped for an agent?
Three questions about the work itself. These fail quietly: nobody notices until the evaluation has nothing to measure against.
The task happens in much the same shape hundreds of times a month.
Repetition pays for the build. Work that is rare, or different every time, never earns back the cost of evaluation and review.
You can write down the full list of actions the agent may take, and the list is short.
An open-ended list means open-ended damage, and no reviewer can approve it. Count the actions before you count the features.
A record of the correct outcome exists somewhere you can test against.
Closed tickets, ledger entries, past decisions. Without a record of the right answer there is no evaluation, and the demo stays the only evidence you have.
Failure containment
Can you survive it being wrong?
Three questions about what happens when the agent makes a mistake. This is the hard gate: a 0 on any question here stops the proof of concept, whatever the total.
The worst single action it can take can be undone, or a human must approve it first.
Refunds, emails, deletes, commitments. The common failure is a tool call that times out instead of failing. The system does not know whether the action happened, retries, and does it twice.
Someone catches a wrong answer before it reaches a customer, a regulator or a ledger.
A wrong answer nobody sees is the expensive kind. If nothing downstream catches it, your evaluation is the only defence, and it will not be enough in week one.
A named person can stop it within minutes, and you can work out afterwards why it acted.
A kill switch nobody owns is not a kill switch. A trace you cannot replay is not an audit trail.
Evaluation evidence
Will you be able to prove it worked?
Three questions about the proof. Decide these before the build. A baseline cannot be measured after the agent has changed the work.
You know what this task costs today: time, money and error rate.
Without a measured starting point, "better" is an opinion. The pilot review becomes an argument about stories.
You can produce the hard cases today: missing information, conflicting history, two near-identical requests at once.
A demo never contains these cases. Production contains them every hour. If you cannot collect them now, you will not collect them under launch pressure.
You can test what it does when a tool returns neither success nor failure.
Timeouts, partial writes, a confident wrong response. Most agent incidents start here, and almost no proof of concept tests it.
Ownership and unit economics
Will the organisation keep it alive?
Three questions about the people and the money. A pilot that never leaves pilot usually has an ownership problem, not a tooling problem.
One named business owner, not a committee, has a number that improves if this works.
If nobody's metric moves, nobody defends the budget at the next planning round. The pilot quietly ages out.
The data it needs is in systems you can call today.
If a data programme has to finish first, that programme is the real project, and it is six months long. Say so now.
Your cost estimate includes tool calls, retries and human review minutes, not only the model bill.
At a hundred times the volume, the model is usually the smallest line. Human review is usually the largest, and it grows in step with volume.
The outcome, read against the rubric.
Answer all 12 questions to read the outcome. Until then this shows the running total, and a blank counts as 0.
Fix first
Lowest scores first, with the move that answers each one before the agent exists.
You have examined one operating decision. Build the wider reasoning through practical work and review in a Living Craft programme. Explore the programmes
Agree these before you score, not after.
A total one point above a line is a narrow, not a yes. The row that matches your result is highlighted.
Four ways to answer most questions before you build the agent.
Listed from cheapest to most expensive. Each one costs less than a pilot that fails in front of a customer.
Write the evaluation set before the spec
Ask for 50 real cases from last quarter with known-correct outcomes. Recorded cases, not invented ones. If the team cannot collect 50 in a week, that is not a scheduling problem. The task has no ground truth, and an agent you cannot grade is an agent you cannot ship. Failing to produce the set is the answer.
Let a person play the agent first
Before building anything, route real requests to a person. They sit behind the same interface the agent would use, follow the same rules, and use the same tools. You learn the real mix of requests, which ones have no clean answer, and whether anyone uses the thing at all. After two weeks you hold a labelled evaluation set you did not have to invent. If the flow is customer-facing, check the disclosure question with legal first.
Break the tools on purpose
In staging, make a tool time out instead of failing. Make it return success for an action that never happened. Make it return the wrong answer, confidently. Chaos engineering has been standard for services for twenty years and is almost unknown for agents. That is odd, because this is exactly where agent incidents start.
Run it in shadow mode
Put it on live traffic and forbid it from acting. Log what it would have done, compare that with what the human did, and review only the disagreements. Those are usually a small share of the cases and hold all of the information. You get a real accuracy number with no risk of harm.
The score is the easy half. The argument is the other one.
A blank scorecard is useful and costs nothing to give away. What it cannot do on its own is settle the rows where two people scored two points apart. That is where the real disagreement about the system lives, and that part takes a room.
Test the idea before you widen the pilot. Explore Architecting Agentic AI Systems: 30 live hours with Sunil Mathew; confirm complete setup and pre-session preparation before joining.
Test the idea before you widen the pilot. Explore Architecting Agentic AI Systems: 30 live hours with Sunil Mathew; confirm complete setup and pre-session preparation before joining. Request a fit call at https://learning.thelivingcraft.ai/programmes/agentic-systems/architecting-agentic-ai-systems/#discuss-architecting-agentic-ai-systems
Published 13 September 2026 · free to use and to pass on