Agent design check

Why it matters An agent is a decision about control flow, not about capability. If the steps are knowable before the request arrives, a workflow does the same job and can be reviewed before it runs. Reaching for an agent because there is a model in the design moves decisions from review time to run time and gets nothing back for it.

In the example Reading an unstructured message and working out which policy applies is open — the next step depends on what the assistant finds. Issuing a refund against an identified order, a policy version and an amount is closed: a fixed sequence, and calling it from a model does not make it anything else.

Why it matters "Handle returns" cannot be checked by anyone. A purpose written as a task with a boundary can be held against the behaviour, and it is what a reviewer measures scope creep against a year later. A goal is only worth writing if it is testable: "safer" is not a goal, "no order is refunded twice" is, because you can go and look.

In the example "Reads a customer’s return request, identifies the order, applies the published returns policy, and recommends or issues a refund within the window that policy defines."

Why it matters Goals tell a reviewer what you were trying to build, which they can usually infer. Non-goals tell them what you decided against, which they cannot — and a non-goal nobody can quite justify is very often the seam where the design is unresolved.

In the example It does not amend an order. It does not cancel a subscription. It does not contact a bank. It does not act on an order it could not identify.

Why it matters An action with no evidence list is an action whose preconditions live in a prompt and in somebody’s memory. Naming them is also what makes it possible for the tool to refuse: a tool cannot check a precondition nobody wrote down.

In the example Before a refund: an identified order; the state of that order; the policy clause that permits it and which version that clause came from; the amount that clause allows; and the absence of a prior refund against the same order.

Why it matters "The policy says returns are accepted within the window" is a recollection sitting in a context window. It cannot be checked after the fact, and it cannot be checked by the tool at the moment of the call. A clause identifier and a version can be both. This is also what makes the system explainable: an answer built out of references is an answer, and an answer built out of the model’s summary of its own reasoning is a story about an answer.

In the example The refund carries clause 4.2 of the returns policy at the version in force that day — not a sentence summarising what the policy seemed to say.

Why it matters Evidence gathered a few steps earlier may have been superseded, and may never have been what the caller says it was. The check that matters is the one at the point where the effect happens, because that is the only check an unusual input cannot route around.

In the example The refund tool re-reads the cited clause at the cited version and re-reads the order state, and refuses if either has moved — rather than accepting the amount it was passed.

Why it matters A broad tool is a permission granted to every path that can reach it, including the paths nobody designed. Narrowness is one of the few properties of a tool that holds regardless of what the model was persuaded to ask for.

In the example "Issue a refund against this order for this amount" rather than "call the payments API".

Why it matters An instruction in a system prompt is a message inside the system. The tool is where the effect happens. Whatever is enforced at the tool holds regardless of what the model was asked to do — and the distinction is structural rather than a claim about how obedient a model is.

In the example The refund tool refuses an amount above what the cited clause permits. The prompt telling the assistant not to exceed the policy is guidance; the refusal is the control.

Why it matters A shared identity gives every tool the union of every tool’s rights and leaves the log unable to answer who did this. The record is also the only thing that can settle a dispute afterwards — "the system decided to" is not evidence.

In the example The refund tool holds refund rights and nothing else. It cannot amend an order, because that is a different tool acting as somebody else.

Why it matters A recommendation is reviewable text. An action moves money, sends a message or changes somebody’s record. Folding them together means one design decision governs both, and the safe version of the first is not the safe version of the second.

In the example The assistant gathers the policy evidence and prepares a recommendation. Issuing the refund is a separate call, with its own controls over what it will allow.

Why it matters A retry, a refresh, and a customer writing in twice all look the same to a system that identifies an action only by the fact that it was asked for. The key has to be derived from the request rather than from the attempt, or the second attempt simply carries a second key.

In the example The same customer sends the same request twice. The refund is keyed to the order and the clause, so the second call returns the first refund rather than paying again.

Why it matters An unknown outcome is not a failure, and treating it as one is how a single action becomes two. Treating it as a success is how somebody is told nothing happened when it did. The design has to be able to go and look.

In the example The refund call times out. Before anything else, the system reads the payment back by its key and finds out whether the money moved.

Why it matters The transcript is what the system said. A review holding only the final answer can tell you the answer was wrong, but not whether the cause was missing evidence, an over-broad permission or a tool returning something unexpected — and those are three different changes. This is the decision that makes the other five improvable.

In the example Why was this refund issued? An answer built from the clause, the version and the tool’s response is an answer. An answer built from the model’s account of its own reasoning is not.

Why it matters Without cases, a change is judged by whether the demo still looks right — which it usually does, because that is the path the demo takes. A failure that does not become a case is a failure you have agreed to have again.

In the example The partially shipped order that was refunded in full is now a case, carrying the evidence the system should have required before acting.

Why it matters The honest output when evidence is missing is not a lower-confidence version of the normal output. It is a different output, and it has to be designed as one — otherwise the system’s only way of expressing doubt is to do the usual thing slightly less well.

In the example The request falls outside any policy the assistant holds. It stops short of the refund and says which fact it could not establish, rather than recommending its best guess.

Why it matters A missing policy, an ambiguous order state and two policy versions that disagree are three situations calling for three different next steps. One threshold throws away exactly the information that would have said which — and a number below a threshold tells the person who receives it nothing about what to do.

In the example "No clause covers this" goes to whoever can decide policy. "Two clauses disagree" goes to whoever owns the policy. "The order state is unclear" is a lookup, not a judgement.

Why it matters "A human is in the loop" is not a design until you can say who, how they are reached, what they see, and what happens to the run while they decide. An intervention path that exists in principle is a queue nobody is watching.

In the example The assistant hands over the request, the order, the clauses it found and the reason it stopped — rather than a conversation for somebody to read from the top.

Why it matters Most of the permission work happens on the far side of the tool boundary, which often means a different team, a different repository and a different review. A permission model that lives only in the agent’s codebase is one integration away from being bypassed, and the first sign of that is nobody being able to name who owns the tool.

In the example The assistant has an owner. So does the refund tool, and it is not the same team.

Why it matters Those three change behaviour without changing anything a test was written against. A prompt edit that widens what the system will attempt is a change to the design, and it should leave the same trace a code change does — otherwise the record of why the system behaves as it does has a hole in it exactly where the behaviour was decided.

In the example Widening the refund tool from "within the policy window" to "within the window, or where a manager has approved" is a design change with a record, not a configuration tweak.