The AI agent tech stack: seven layers

The seven layers

Most published stack diagrams stop at four. That is not an oversight — the first four layers are genuinely all you need to produce something that works, and they are what every framework competes on. The last three, and particularly the last two, are what you discover you need at the exact moment an agent touches something real.

The organising principle is replaceability. Each layer should be swappable without touching the others, and the test is concrete: if changing your model provider means rewriting tool definitions, or adding an approval gate means editing the control loop, the boundaries are in the wrong place.

1. Model

The mistake here is not choosing the wrong model; it is choosing anything at all in a way that cannot be revisited. Model releases currently outpace deployment cycles, so a system that names a model in forty places has already lost a decision it will need to make again within months.

Per-task, not per-system

A single model everywhere is almost always wrong in one direction or the other. Planning, ambiguity resolution and anything with a wide solution space want capability. Mechanical extraction, classification and routine tool selection do not. The practical shape is a capable model on the main thread and cheaper models for delegated work that is well-specified — see the Agent SDK guide for how per-subagent model selection works in a real harness.

One constraint worth knowing before you design around model switching: prompt caches are model-scoped. Changing model mid-conversation invalidates the cache, so the cheap pattern is a stable main thread with variety pushed into subagents rather than a router that swaps models turn by turn.

2. Context

Four things compete for the window, and they have very different value density. Instructions are small and load-bearing. Tool schemas are moderate and mostly static. History grows without bound. Retrieved content is the largest and the most variable in usefulness.

Caching is a prefix match, and that dictates layout

Prompt caching keys on the exact bytes of the prompt up to a breakpoint, which means one changing byte early invalidates everything after it. That single fact determines the whole layer’s design: stable content first, volatile content last, and nothing dynamic interpolated into the system prompt.

The classic self-inflicted wound is a timestamp or a session ID in the system prompt. It sits at position zero, changes every request, and quietly makes the entire conversation uncacheable. If your cache-hit rate is zero across identical-looking requests, look for something like that before looking anywhere else.

3. Tools

Two design decisions in this layer matter more than the rest of the stack combined.

Granularity: bash versus dedicated tools

A shell tool gives an agent enormous reach and gives your harness nothing: every action arrives as an opaque command string with the same shape. Promoting an action to a dedicated tool with a typed schema gives you something you can gate, render, log, rate-limit and parallelise.

Start with breadth and promote what you need to control. The signal that an action should be promoted is that you want to do anything to it other than run it.

Guarantees: exactly-once is not free

Every tool that changes state needs an explicit answer to “what happens if this runs twice?”. Agents retry on their own initiative and often cannot tell whether a previous attempt succeeded, so this is not a theoretical concern — it is the most common source of real damage in agent deployments. The mechanism differs from the HTTP pattern because an agent’s retry is not a byte-identical request; see idempotency for agent actions for how to key on intent instead.

4. Control loop

This is the layer people write themselves and shouldn’t. The loop is not where differentiation lives, it is well-solved by several harnesses, and hand-rolling it means owning a class of bug — malformed tool-result ordering, dropped parallel calls, mishandled pause states — that has nothing to do with your product.

// Bounds — every one of these should be configurable per run.
maxTurns:      40,        // agentic iterations before a hard stop
maxBudgetUsd:  5,         // spend ceiling, enforced
taskBudget:    { total: 64_000 },  // budget the model is AWARE of, so it paces itself
timeoutMs:     15 * 60_000,

// Stop conditions — distinct outcomes, not one boolean.
//   completed  → objective met
//   exhausted  → hit a budget; partial work may be salvageable
//   escalated  → hit an authority boundary; waiting on a human
//   failed     → unrecoverable

The distinction in that caption is worth internalising. A hard output ceiling truncates the model mid-thought with no warning. A budget the model can see lets it prioritise and finish gracefully. Production loops want both: the budget for behaviour, the ceiling as a backstop.

Loops compound errors

A single wrong tool call is bounded. A loop is not: the agent acts on a mistaken belief, observes a world that now reflects its mistake, concludes something further, and acts again. By step six the original error is buried under five steps that each followed logically from it, and the transcript reads as reasonable the whole way down. That is why the stop conditions above need to be four distinct outcomes rather than success or failure — escalated in particular is not a failure, and treating it as one is how agents get retried into damage.

5. Memory

Three distinct things get called memory, and conflating them is why the layer is usually built badly.

The third is the highest-leverage and the least commonly built. An agent that solves a multi-step problem, writes down the procedure, and reuses it next time is strictly cheaper and more consistent than one that re-reasons from scratch. It is also the design decision behind the fastest-growing open agent runtimes, which is a reasonable signal that it works.

Memory is an injection surface

Anything the agent writes and later reads as instruction is a persistence channel for a prompt injection. Content absorbed from an attacker-controlled source in one session and written to memory affects every later session that loads it — a compromise that outlives the run that caused it. Treat memory writes as untrusted on read, and never store credentials there: memories are replayed verbatim into future contexts.

6. Governance

That is the whole layer in one sentence. A constraint written into the system prompt is a preference; the attacker, or simply an unusual situation, gets to make the closing argument. A constraint enforced at the tool-call boundary is a rule.

The four controls

Two of those have a failure mode worth naming. Least privilege is usually skipped because permissions are inherited rather than designed — the agent gets wired to an integration user that already had broad access and nobody narrowed it, because the demo runs either way. And a gate that fires constantly gets clicked through, which converts a control into a formality that produces a record implying review that did not happen.

Deciding where the boundary sits is its own problem, and it is more a question about the organisation than about the software — when should an agent ask covers the three signals worth gating on. For the adversarial view of the same layer, see the agent threat model.

7. Evidence

This layer is usually confused with logging, and the difference is the audience. Logs are for you, during debugging, and can be lossy. Evidence is for someone who was not there — a reviewer, an auditor, a customer asking why — and it has to be complete enough to answer a question nobody anticipated.

What a trace needs to contain

Enough to answer “why did it do that?” without re-running anything: the objective, each action with its arguments, the result returned, the reasoning that selected it, which authority rule allowed it, and who approved anything gated. The last two are what makes it an audit trail rather than a transcript.

Evaluation is a trajectory problem

Reviewing outputs is sufficient for a system that only produces text. For an agent it is not: the same final answer can be reached through an acceptable path or an alarming one, and only one of those should pass. You are evaluating the sequence of decisions, not the answer — which means your eval harness needs traces, which means this layer is a prerequisite for improving anything above it.

That dependency is the strongest practical argument for building the evidence layer early rather than last. Without traces you cannot build evals; without evals every change to the layers above is a guess.

What to build first

Layer one barely appears in that list, which is deliberate. Model choice feels like the first decision and is nearly the last one that matters — as long as you have kept it swappable, which costs almost nothing at the start and a rewrite later.

The through-line: the first five layers make an agent that works, and they are the ones with the most tooling, the most tutorials and the most vendor attention. The last two make an agent you can put in front of someone, and you will mostly have to build them yourself. That asymmetry is not a gap in the ecosystem — it is where the actual work is.

If you are deciding what to build on rather than what to build, the Agent SDK guide covers one harness in production detail, and the definitions piece is worth reading first if it is not yet settled whether you need an agent at all.