Most agent demos start with the model. That makes sense, because the model is the part that feels like magic, but it is not where the practical work ends up.
The model does matter. I use Claude and Codex heavily, and I am learning Microsoft’s agent stack because that is where so many businesses already live. Still, the more I build, the more it seems that the model is the part most likely to change underneath me. Whatever I lean on today will be cheaper, faster or simply replaced before long. So the patterns I care about are the ones that survive a change of model.
When I strip an agent system back to what actually keeps it useful, I keep coming back to the same six layers. The first version of this post just listed them. Since then I have kept running my own systems, at home and inside a real business, and each layer has earned a story. So here is the list again, this time with the receipts.
1. Memory: what the agent can reliably know
Memory is not the model’s training data, and it is not the chat transcript. It is the set of facts the agent can look up and trust: what was decided, what broke last time, what the fix was, and which system owns the truth.
The example that sold me was small. A media server at my place died after every power outage, and my home agent spent a long session tracing it to a race between the file server coming up and the container that needed it. The fix was fine. The valuable part was the sentence written into the agent’s memory afterwards: what the symptom looks like, what the actual cause is, and what to do. The next outage was a lookup, not an investigation.
That is the pattern. Work becomes knowledge only if something writes it down where the next run will find it.
The failure mode when this layer is missing: an agent with no memory re-learns your business every morning, and an agent with wrong memory repeats yesterday’s mistake with total confidence. Both look busy. Neither compounds.
2. Tools: what the agent is allowed to touch
An agent’s tools define its blast radius. The temptation is to hand over broad access, a database connection, an admin credential, because it makes the demo work faster. That is how you build a mystery box with root access.
The system I built for a distribution business never touches the ERP directly. It works through a small set of purpose-built capability servers that expose narrow, safe operations, read-only by default. The agent can ask for a customer’s order history or for products with cost drift. It cannot ask for a SQL prompt. That boundary is most of the reason the whole thing is trustworthy enough to run against live business data.
The failure mode: when something goes wrong inside a broad-access agent, you cannot say what it could have done, only what you hope it did. Narrow tools turn that question into a checklist.
3. Workflow: the sequence of work that actually needs to happen
The work had a shape before AI arrived. An order comes in, gets checked against what the customer actually buys, the totals get sanity-checked, and only then does anything get posted. The agent’s job is to run that shape, not to improvise a new one each time.
So the workflows worth having are packaged: written down once, with their steps, their checks, and their definition of done, and then run the same way every run. In my systems these are skills, and the recurring ones are wired to schedules so nobody has to remember to fire them.
Most of the time, when someone tells me their agent “got it wrong”, we find that nobody had written down what right was. The model gets blamed for a missing spec.
4. Approvals: where a human must stay in the loop
Not all actions are equal. Reading is free. Internal drafts are cheap. Writes to live business data, money movements, and anything that leaves the building to a real person are a different category, and the system should know the difference structurally, not by vibes.
The rule that has survived everywhere I have applied it: consequential writes run dry-run first, show exactly what they intend to change, and execute only on explicit approval. My own agent publishes to my website on its own, because the cost of a bad publish is a quick revert. It does not send an email to a human being without me seeing it, because you cannot revert a sent email.
The failure mode: either no gates, which eventually produces the incident that kills the whole project’s credibility, or gates on everything, which produces an agent so throttled nobody uses it. The design work is deciding which actions deserve which.
5. Audit: what happened, why, and who signed off
Trust comes from the audit trail, not from hoping the model behaved.
Every automated write in my business system lands in a log: what changed, what triggered it, what the dry-run showed, who approved it. When a number looks wrong, “what touched this” has an answer measured in seconds. And the same trail is what turns reliability from a feeling into a figure. I can say the scheduled jobs run at roughly 99.7% success because the log says so, not because the system seems fine.
The failure mode: an agent system without an audit trail is unfalsifiable. When it works you cannot prove it, and when it fails you cannot explain it. Either way you are asking people to trust vibes, and rightly, they will not.
6. Escalation: when the system should stop and ask
My favourite example is the smallest. For a live conference demo, I had an agent publish a photo from my phone to a blog post on stage. The publish script had one guard: if no fresh photo had arrived, refuse, and tell me plainly to send one, rather than grabbing the most recent file it could find, which would have been a stale rehearsal shot. The guard existed because that exact failure had already happened once.
That is escalation in miniature. The system knows the one condition under which it must not proceed, and its job at that point is to stop and say so clearly. Not retry silently. Not improvise around the guard. Stop, and ask.
The failure mode is an agent that treats every obstacle as something to route around. That trait looks like initiative in a demo and becomes terrifying in production.
The boring layer decides everything
None of this is glamorous. It is the plumbing nobody puts in a demo, and it is the layer that quietly decides whether an agent becomes infrastructure you trust or another clever mess you clean up after.
It is also, conveniently, the layer that survives. Models keep changing underneath me, and these six things have not moved.
If you are building in this space too, I would still love to hear which of these layers you find hardest in practice. For me it has been memory, by a distance. Feel free to reach out.