A checklist for choosing a model — for one step, not for the whole system.
Most model choices are made on a benchmark, a price and a demo. None of those tell you what a model leaves out, and what it leaves out is what costs you. This is what to check instead, in the order to check it.
26 checks · 4 test cases you build yourself · one afternoon
The boxes tick in your browser and nothing is saved. Nothing is stored on this site, nothing is sent anywhere, and a refresh clears them. Print the page, or save it as a PDF, if you need to keep a filled copy.
Every item below says the same three things: what it is, what a pass looks like, and what a fail costs you. The last one is why the item is on the list.
Four rules. Break one and the rest stops meaning much.
- 1 Choose one step, not the whole system
A model is staffed on a step: read this and pull out the fields, draft this reply, decide whether this goes to a person. A system has many steps and they do not all need the same model. Run this checklist once per step.
- 2 Do the gates before you test anything
The gates are read off the model card and the contract. They take about half an hour and they usually remove half your list. Testing a model that cannot be deployed where you need it is time you do not get back.
- 3 Write your answer key before the first run
List what a correct output must contain, on paper, before you see any output. A key written afterwards is written around what the model happened to produce, and it will never show you what is missing.
- 4 Same cases, same prompt, same permissions, ten runs each
Change one thing at a time and only the candidate. Write down the model name and the exact version beside every result. One run tells you what it can do. Ten tell you what it does.
Write down the step before you look at any model.
Fill these in first. A model that drafts something a person will read can afford mistakes that the same model cannot afford when it acts alone, so the weights further down change completely depending on what you wrote here.
The step this model would do
It is allowed to
It is not allowed to
It must hand over to a person when
Pass or fail, from the model card. No score rescues a fail.
Half an hour of reading, before you run a single test. It usually removes half the list.
A model card gets a candidate into the room. It never wins the job. A published benchmark has never told anyone whether a model can do their specific work.
Score each one 1 to 5, on the same cases, for every candidate.
| What you score | The question you are answering | Drafts for a person | Flags and routes | Acts on its own |
|---|---|---|---|---|
| Accuracy | On the cases where you know the answer, did it get them right? | 3 | 3 | 3 |
| Things it left out | How many required items never appeared in the output at all? | 3 | 5 | 5 |
| Saying "I don’t know" | When something is missing, does it say so, or does it fill the gap? | 2 | 4 | 5 |
| Staying inside permissions | Does it ever try to do something you told it not to do? | 1 | 2 | 5 |
| Raising what is about to expire | Does it escalate a closing deadline, or quietly let it pass? | 1 | 5 | 5 |
| Consistency | Same case, ten runs. How many times did it get the hard one? | 2 | 4 | 5 |
| How obvious a mistake is | When it is wrong, do you see it in five seconds or in five minutes? | 4 | 4 | 3 |
| Cost per case solved | Not cost per token. Include retries and the minutes a person spends fixing it. | 3 | 3 | 2 |
| Clean tool calls | Malformed fields, invented names, arguments that do not match the schema. | set it yourself | set it yourself | set it yourself |
| Long inputs | Accuracy on the part near the end of a long input, not the first page. | set it yourself | set it yourself | set it yourself |
| Speed at the slow end | Measured on a real, long input rather than a short test one. | set it yourself | set it yourself | set it yourself |
| Effort to run | Retries, human corrections, and how easily the prompt breaks when inputs change. | set it yourself | set it yourself | set it yourself |
Two rows are marked. They are the two that a quality review will not raise, because neither of them puts anything wrong on the page. They are marked for that reason, not because they outrank the others.
Pick the column that matches what you wrote in part 0. A model that drafts for a person can afford to miss more than a model that acts alone. Four rows carry no starting number, because what they are worth depends on the job you are putting them in. Write your own.
Any one of these ends a candidate, whatever it scored.
These four end a candidate whatever it scored, because they are the mistakes you are least likely to notice and the cost has no ceiling. An average cannot price a mistake that hides itself.
Four cases, built out of your own work.
Use real inputs with the names removed. A public benchmark cannot tell you how a model behaves on the documents your team actually receives.
The ordinary case
How to build it Take a normal piece of work with nothing unusual in it. Change nothing.
Right answer Everything found, nothing invented.
What it tells you Very little. Every candidate passes it. Run it anyway, because a candidate that fails here has saved you the rest of the afternoon.
The missing input
How to build it Take a normal case and remove one thing the input points at: an attachment, a linked record, a prior message. Leave the reference in place.
Right answer It says what was referenced and not supplied, and produces nothing in its place.
The tempting failure A confident, tidy answer assembled from the surrounding context.
The planted item
How to build it Take a real case and move one important item somewhere nobody looks: the end of a long document, an attachment, a footnote, the last message in a thread. Change nothing else. Write down what you moved.
Right answer The item appears in the output.
The failure A clean, correct, well-organised answer that never mentions it.
The closing deadline
How to build it Set the clock so an action is about to be due, or has just become overdue. Let the candidate flag and draft. Do not let it sign, commit or spend.
Right answer Marked urgent, drafted, addressed to a named person, with the date arithmetic shown.
The failure Marked low priority, or "monitor". It breaks no rule. It costs you the deadline by doing nothing.
Permission systems control what a model may do. Very few control what it may let expire.
The same three cases, in four different jobs
Illustrations of how to build a case. They describe no real team, product or system.
| The job | Missing input | Planted item | Closing deadline |
|---|---|---|---|
| Support ticket triage | The ticket refers to an earlier case number that is not in the thread. | The customer states their account tier in the fifth message, not the first. | The response is due under a service agreement in two hours. |
| Invoice and payment checks | The invoice cites a purchase order that was not attached. | A delivery charge sits on the last line of a second page. | An early-payment discount expires tomorrow. |
| Code review | The change refers to a migration file that is not in the diff. | A permission check is removed in a file nobody expects it in. | A dependency with a published end-of-support date is being pinned. |
| Insurance claim intake | The claim form names a medical report that was never uploaded. | An exclusion appears in an endorsement rather than in the policy body. | The notification window for the claim closes this week. |
Case 1 is not in the table because an ordinary case looks the same in every job. Build it anyway. It is the one that tells you a candidate has failed before you spend the afternoon.
The step most teams skip, because nothing is on the page to mark wrong.
- 1 Write the answer key before you run anything. List what a correct output must contain. This only works against a key you wrote yourself.
- 2 For every run, mark each item on the key as found, partly found or missing. Ignore how well it is written.
- 3 Missed rate = missing items ÷ total items on the key, counted across all ten runs.
- 4 Note which item was missed, not only how many. A candidate that always misses the thing at the end has a position problem, and that is different from random noise.
- 5 Separately, hand one output to a colleague who has not seen the key and time how long they take to notice the gap. Under 10 seconds means the mistake is obvious. Over a minute means, in practice, it ships.
Keep steps 2 and 5 apart. Being right and being obviously wrong are two different properties, and the second is the one most teams never measure.
One page. Written when you decide, not afterwards.
Print this section to fill it in, or copy the headings into your own record.
The step
The step, in one line
Allowed to
Not allowed to
Must hand over when
Date
Decided by
What was tested
Candidates — name, exact version, how deployed
Failed a gate — which candidate, which gate
Cases run — how many, and how many runs each
Answer key written by
What was decided
Disqualifiers hit — which candidate, which one, evidence
We chose
Because — one sentence, naming the thing that decided it
Only for — input length, work types, volume, languages
Weaknesses we are accepting
What covers those weaknesses
Results
One row per candidate. Nothing on this page adds them up.
| Accuracy | Left out | Says "don’t know" | Permissions | Deadlines | Consistency | Time to spot error | Cost per case | Slow-end speed | |
|---|---|---|---|---|---|---|---|---|---|
| A | |||||||||
| B | |||||||||
| C |
Re-test if any of these happen
Next re-test due
This page is why a deprecation notice becomes a Tuesday instead of a project. The model is not the asset you keep. This is.
Four hours and five minutes, in this order.
-
Write down the step, and what it is and is not allowed to do.
-
Run the gates against each candidate model card.
-
Write the answer key for the cases you are about to build.
-
Build the four cases out of your own work.
-
Run every surviving candidate ten times per case.
-
Score the missed items against the key.
-
Time a colleague spotting the error without the key.
-
Fill in the decision record.
If the result surprises you, the checklist is working.
This picks a model. It does not tell you the system holds up.
Everything you scored is a claim about one step on one day. Keeping that claim true needs an eval set you run again when the version moves, and a register that catches the work the model never mentions.
The Living Craft runs a 6-week cohort on exactly this: building agents for decisions where the right answer is genuinely debatable, and building the evals that show they hold up. 8 seats, by application only, for Staff+ engineers, engineering managers and architects who ship to production.
See the cohort → All resources →
Published 15 September 2026 · free to use, to change and to pass on