Designing agentic systems

This is written for engineers, architects and engineering leaders who are about to put a language model behind something that acts, or who have been handed someone else’s design and asked whether it is sound.

The six decisions start to matter at one specific moment: when the software stops recommending and starts doing. A person reviewing a recommendation provides a control before an action. When a system can call tools that change the world, those tool boundaries need their own controls. Human approval must be designed into that path when it is required.

Use the guide before a build or during a review. Its focus is systems that can act. A read-only assistant still needs data-access and output controls. A prototype also deserves these questions before anyone proposes giving it authority.

This is an illustrative example. It is not an account of a real customer system, and no incident is being described.

Imagine a returns assistant. A customer writes in about a return. The assistant finds the relevant policy, looks up the order, and recommends a refund. The conversation reads well. You can follow the reasoning, and you can see how it would save somebody a lot of repetitive work.

Then someone asks a perfectly reasonable question: can it issue the refund as well?

On a diagram that is one more connection. In the world, the system can now move money.

That single change turns a set of comfortable questions into uncomfortable ones. What gives it permission? Which evidence is enough? If the same customer sends the same request twice, do we pay twice? If the refund tool times out and we cannot tell whether the money moved, what does the assistant do next? And if the policy does not cover the case in front of it, who decides?

None of those is an advanced topic. They are the design. A demo makes the happy path easy to see; the design is the part that decides what happens on every other path.

1. Does this need an agent at all?

Choosing an agent gives the system scope to select its next step at run time instead of following a sequence fixed in advance. That is worth doing when the next step genuinely depends on what the system finds, and it is a poor trade when you already know the steps and were only reaching for an agent because there is a model in the design.

For the returns assistant, part of the work is closed: given an identified order, a policy version and an amount, issuing a refund is a fixed sequence. Part of it is open: reading an unstructured message, working out which of several policies applies, noticing that a shipment was partial. The open part is a candidate for an agent. The closed part is a function, and calling it from a model does not make it one.

When a workflow is enough works through the test in detail, including what you give up when you move a decision from review time to run time.

2. What is the system for, and where does it stop?

Write the purpose as a task and a boundary, in a sentence somebody else can check. For example: “reads a customer’s return request, identifies the order, applies the published returns policy, and recommends or issues a refund within the window that policy defines.”

Then write the non-goals, and treat them as part of the design rather than as a disclaimer. It does not amend an order. It does not cancel a subscription. It does not contact a bank. It does not act on an order it could not identify.

I have found the non-goals more useful in a review than the goals. The goals tell a reviewer what you were trying to build, which they can usually infer. The non-goals show the choices they cannot infer. A non-goal that nobody can justify often marks an unresolved decision.

A goal is only worth writing if it is testable. “Safer” is not a goal. “No order is refunded twice” is a goal, because you can go and look.

3. What evidence must it hold before it acts?

For each action, name the facts that must be established first, and where each one comes from.

For issuing a refund, that might be: an identified order; the state of that order; the policy clause that permits the refund, and which version of the policy it came from; the amount that clause allows; and the absence of a prior refund against the same order.

The important word is established. Evidence is a reference, not a recollection. “The policy says returns are accepted within the window” is a paraphrase sitting in a context window; it cannot be checked afterwards and, more importantly, it cannot be checked by the tool at the moment of the call. A clause identifier and a version can be. The difference sounds pedantic until the policy changes and you need to know which version each decision was made under.

These references also make a decision easier to examine. When someone asks why a particular refund happened, an answer built out of references is an answer. An answer built out of the model’s summary of its own reasoning is a story about an answer.

4. What may each tool do?

Enforce authority at the tool boundary.

An instruction in the system prompt saying the assistant must not refund more than the policy allows is guidance. A refund tool that refuses amounts above what the cited clause permits is a control. The distinction is structural rather than a claim about how obedient a model is: the prompt is a message inside the system, and the tool is the place where the money actually moves. The tool can reject a disallowed request regardless of what the model was persuaded to ask for. Its checks still need testing.

So each tool gets its own narrow authority: the smallest scope that lets it do its one job, limits it enforces itself, an identity it acts under, and a record of every call.

Who may call the tool covers how to draw those boundaries, including the case where the right permission is “a named person approves this one.”

5. What does it do when it is uncertain?

Most designs have two outcomes: act, or fail. Systems that behave well under uncertainty have three: act, refuse, and hand over with the specific gap named.

The returns assistant will meet requests the policy does not cover, orders whose state is ambiguous, and two policy versions that disagree. Those are different situations and they deserve different handling, but they share one property: the honest output is not a lower-confidence version of the normal output. It is a different output.

What to do with uncertain evidence works through the kinds of uncertainty and what each one should route to.

6. What can you see afterwards?

Keep a record of each run that holds the task, the evidence used with its references, every tool call with its arguments and its result, the points where the system chose between options, and the outcome.

Keep more than the conversation transcript: the review needs what the system used and what it did.

Two reasons. The first is that you cannot evaluate what you cannot see: a review that only has the final answer can tell you the answer was wrong, but not whether the cause was a missing policy, an over-broad permission or a tool that returned something unexpected. Those three call for three different changes. The second is that a dispute about a refund is settled by evidence, and “the system decided to” is not evidence.

This is the decision that makes the other five improvable. Once the record shows which evidence was used and which tool did what, a failure points at the specific decision to revisit.

I want to be plain about the basis for the above, because a confident tone is easy to produce and worth very little on its own.

This is design reasoning, not a measured result. There is no benchmark in this guide and no claimed effect size, because I do not have one I would stand behind. Where an argument would need evidence I do not have, I have left the claim out rather than dressed it up.

What the reasoning does rest on is two structural facts that hold regardless of which model you use.

The first is that a control has to sit where the effect happens. The tool call is the moment the world changes. Anything enforced there is enforced; anything expressed only as an instruction earlier in the system is a request, and it competes with everything else in the context, including text that arrived from outside. That is why decisions three and four are separate items rather than one, and why they both sit above the model in the ordering.

The second is that the purpose boundary shapes the rest of the design. If it changes, the evidence, tools and evaluation may need to change with it. A model swap also needs relevant evaluation, even when the integration is straightforward. Making the expensive decisions first, and writing them down where a reviewer can argue with them, is the whole reason for the ordering.

There is a third, softer argument. These six are close to what a good reviewer asks for anyway. If the answers exist before the review, the review can be about the trade-offs. If they do not, the review is spent excavating the design, and everybody leaves having discussed less than they wanted to.

This is a design checklist, not a correctness argument. Answering the six questions makes the proposed boundaries and records available for review. It does not establish that they work, and it would be a bad idea to read a completed checklist as a readiness certificate.

It is also incomplete on purpose. Three areas are only touched here: evaluating behaviour rather than answers; the reliability questions around retries, duplicate actions and unknown outcomes; and the full cost of running the thing, which is rarely the price of a response. Each deserves its own treatment and none of them is covered by the six decisions above.

Two more honest gaps. If the content your agent reads can be written by someone who benefits from the action (a customer’s own message, a supplier’s document or a web page), then the evidence question is harder than section three makes it sound, because some of the input is adversarial by construction. The design idea holds; the sourcing rules have to get stricter, and a permission that assumed cooperative input has to be re-examined.

And none of it survives without a named owner. A boundary that everybody agrees with and nobody owns is renegotiated, quietly, by whoever is under the most delivery pressure that month.

If you have a design in front of you, the useful next step is to answer the six questions against it in writing and see which ones you cannot answer yet. That is usually more informative than the ones you can.

The Agent Design Check walks the same ground as a structured checklist you can run in the browser, and tells you which questions are still unanswered rather than scoring you.

The three decisions that carry the most weight have a guide of their own, and they are linked at the foot of this page.

• When a workflow is enough: locate the part of the task that needs an open choice.
• Who may call the tool: put limits at the point where an action happens.
• What to do with uncertain evidence: give missing, conflicting and incomplete evidence an explicit route.

Use the Agent design canvas to write down the six decisions for your system. If you want live guidance as you build and review it, compare the Agentic Systems programmes.