Agent design check

Why it mattersAn agent is a decision about control flow, not about capability. If the steps are knowable before the request arrives, a workflow does the same job and can be reviewed before it runs. Reaching for an agent because there is a model in the design moves decisions from review time to run time and gets nothing back for it.

In the exampleReading an unstructured message and working out which policy applies is open — the next step depends on what the assistant finds. Issuing a refund against an identified order, a policy version and an amount is closed: a fixed sequence, and calling it from a model does not make it anything else.

Why it matters"Handle returns" cannot be checked by anyone. A purpose written as a task with a boundary can be held against the behaviour, and it is what a reviewer measures scope creep against a year later. A goal is only worth writing if it is testable: "safer" is not a goal, "no order is refunded twice" is, because you can go and look.

In the example"Reads a customer’s return request, identifies the order, applies the published returns policy, and recommends or issues a refund within the window that policy defines."

Why it mattersGoals tell a reviewer what you were trying to build, which they can usually infer. Non-goals tell them what you decided against, which they cannot — and a non-goal nobody can quite justify is very often the seam where the design is unresolved.

In the exampleIt does not amend an order. It does not cancel a subscription. It does not contact a bank. It does not act on an order it could not identify.

Why it mattersAn action with no evidence list is an action whose preconditions live in a prompt and in somebody’s memory. Naming them is also what makes it possible for the tool to refuse: a tool cannot check a precondition nobody wrote down.

In the exampleBefore a refund: an identified order; the state of that order; the policy clause that permits it and which version that clause came from; the amount that clause allows; and the absence of a prior refund against the same order.

Why it matters"The policy says returns are accepted within the window" is a recollection sitting in a context window. It cannot be checked after the fact, and it cannot be checked by the tool at the moment of the call. A clause identifier and a version can be both. This is also what makes the system explainable: an answer built out of references is an answer, and an answer built out of the model’s summary of its own reasoning is a story about an answer.

In the exampleThe refund carries clause 4.2 of the returns policy at the version in force that day — not a sentence summarising what the policy seemed to say.

Why it mattersEvidence gathered a few steps earlier may have been superseded, and may never have been what the caller says it was. The check that matters is the one at the point where the effect happens, because that is the only check an unusual input cannot route around.

In the exampleThe refund tool re-reads the cited clause at the cited version and re-reads the order state, and refuses if either has moved — rather than accepting the amount it was passed.

Why it mattersA broad tool is a permission granted to every path that can reach it, including the paths nobody designed. Narrowness is one of the few properties of a tool that holds regardless of what the model was persuaded to ask for.

In the example"Issue a refund against this order for this amount" rather than "call the payments API".

Why it mattersAn instruction in a system prompt is a message inside the system. The tool is where the effect happens. Whatever is enforced at the tool holds regardless of what the model was asked to do — and the distinction is structural rather than a claim about how obedient a model is.

In the exampleThe refund tool refuses an amount above what the cited clause permits. The prompt telling the assistant not to exceed the policy is guidance; the refusal is the control.

Why it mattersA shared identity gives every tool the union of every tool’s rights and leaves the log unable to answer who did this. The record is also the only thing that can settle a dispute afterwards — "the system decided to" is not evidence.

In the exampleThe refund tool holds refund rights and nothing else. It cannot amend an order, because that is a different tool acting as somebody else.

Why it mattersA recommendation is reviewable text. An action moves money, sends a message or changes somebody’s record. Folding them together means one design decision governs both, and the safe version of the first is not the safe version of the second.

In the exampleThe assistant gathers the policy evidence and prepares a recommendation. Issuing the refund is a separate call, with its own controls over what it will allow.

Why it mattersA retry, a refresh, and a customer writing in twice all look the same to a system that identifies an action only by the fact that it was asked for. The key has to be derived from the request rather than from the attempt, or the second attempt simply carries a second key.

In the exampleThe same customer sends the same request twice. The refund is keyed to the order and the clause, so the second call returns the first refund rather than paying again.

Why it mattersAn unknown outcome is not a failure, and treating it as one is how a single action becomes two. Treating it as a success is how somebody is told nothing happened when it did. The design has to be able to go and look.

In the exampleThe refund call times out. Before anything else, the system reads the payment back by its key and finds out whether the money moved.

Why it mattersThe transcript is what the system said. A review holding only the final answer can tell you the answer was wrong, but not whether the cause was missing evidence, an over-broad permission or a tool returning something unexpected — and those are three different changes. This is the decision that makes the other five improvable.

In the exampleWhy was this refund issued? An answer built from the clause, the version and the tool’s response is an answer. An answer built from the model’s account of its own reasoning is not.

Why it mattersWithout cases, a change is judged by whether the demo still looks right — which it usually does, because that is the path the demo takes. A failure that does not become a case is a failure you have agreed to have again.

In the exampleThe partially shipped order that was refunded in full is now a case, carrying the evidence the system should have required before acting.

Why it mattersThe honest output when evidence is missing is not a lower-confidence version of the normal output. It is a different output, and it has to be designed as one — otherwise the system’s only way of expressing doubt is to do the usual thing slightly less well.

In the exampleThe request falls outside any policy the assistant holds. It stops short of the refund and says which fact it could not establish, rather than recommending its best guess.

Why it mattersA missing policy, an ambiguous order state and two policy versions that disagree are three situations calling for three different next steps. One threshold throws away exactly the information that would have said which — and a number below a threshold tells the person who receives it nothing about what to do.

In the example"No clause covers this" goes to whoever can decide policy. "Two clauses disagree" goes to whoever owns the policy. "The order state is unclear" is a lookup, not a judgement.

Why it matters"A human is in the loop" is not a design until you can say who, how they are reached, what they see, and what happens to the run while they decide. An intervention path that exists in principle is a queue nobody is watching.

In the exampleThe assistant hands over the request, the order, the clauses it found and the reason it stopped — rather than a conversation for somebody to read from the top.

Why it mattersMost of the permission work happens on the far side of the tool boundary, which often means a different team, a different repository and a different review. A permission model that lives only in the agent’s codebase is one integration away from being bypassed, and the first sign of that is nobody being able to name who owns the tool.

In the exampleThe assistant has an owner. So does the refund tool, and it is not the same team.

Why it mattersThose three change behaviour without changing anything a test was written against. A prompt edit that widens what the system will attempt is a change to the design, and it should leave the same trace a code change does — otherwise the record of why the system behaves as it does has a hole in it exactly where the behaviour was decided.

In the exampleWidening the refund tool from "within the policy window" to "within the window, or where a manager has approved" is a design change with a record, not a configuration tweak.