Semantic caching for AI agents

What semantic caching is

The motivation is sound. Exact-match caching performs poorly on natural language because users phrase the same question forty ways, so hit rates stay near zero. Semantic matching is the obvious fix, and for FAQ-shaped traffic it works well — reported gains for the pattern run to an order of magnitude on cost and considerably more on latency.

The architecture is consistent across implementations, and worth knowing because each component is a place the design can go wrong: an embedding generator to vectorise the request, a vector store to find the nearest past requests, a cache storage layer holding the responses, a similarity evaluator deciding whether a candidate counts as a hit, and a cache manager handling eviction.

Three kinds of caching, three jobs

Conflating the first and third is the most common mistake in this area, and it leads teams to build a semantic cache while their prompt cache is silently broken — which is both harder and far less valuable than fixing the prompt cache.

Fix that first. It is a one-line change with no accuracy risk, and it frequently delivers more than a semantic cache would.

Why it usually fires on the wrong layer

Picture the actual sequence. Turn one is the objective. Turn two is the objective plus a tool call plus its result. Turn three adds two more. By turn six the request is substantially unlike anything that came before it in the same run, let alone in a different one.

There is a second, worse problem. Even where similarity does cross the threshold, the thing being cached is a decision about what to do next — and the correct next action depends on details the embedding flattens away. Two nearly identical states can require different actions, and a cache that cannot tell them apart will confidently supply the wrong one. A near-miss on a factual answer is a wrong sentence. A near-miss on a tool selection is a wrong action.

Where it does work

Semantic caching earns its place on the edges of an agent system rather than inside the loop:

Cache tool results instead

This is the recommendation this article exists to make. It is less interesting than semantic caching and considerably more effective.

Read-only tools are the target: a document fetch, a schema lookup, a reference-data query. Key on the tool name plus the canonicalised arguments, set a TTL that reflects how fast the underlying data actually changes, and you get most of the available saving with none of the similarity exposure.

// Only read-only tools. Only if the tool declares itself cacheable.
function cacheKey(tool: string, args: Record<string, unknown>, scope: AuthScope) {
  // Canonicalise: sorted keys, so {a,b} and {b,a} are one entry.
  const canonical = JSON.stringify(args, Object.keys(args).sort());
  // Scope is part of the key — see the authorization section below.
  return `tool:${tool}:${scope.tenantId}:${scope.principalId}:${sha256(canonical)}`;
}

async function callTool(tool: ReadOnlyTool, args: Args, scope: AuthScope) {
  const key = cacheKey(tool.name, args, scope);
  const hit = await cache.get(key);
  if (hit) return hit;

  const result = await tool.run(args);
  await cache.set(key, result, { ttlSeconds: tool.cacheTtlSeconds });
  return result;
}

The key must carry the authorization scope

This is the failure that turns a performance optimisation into an incident, and it is absent from most writing on the subject because most writing on the subject assumes a single-tenant FAQ bot.

The mechanism is unglamorous. User A at Acme asks “what’s our outstanding balance?”. User B at Globex asks the same words. The embeddings are identical — the sentences areidentical — so the second request hits the cache and receives Acme’s balance. No vulnerability was exploited. The cache simply did what it was built to do.

There is a real tension here and it should be named: every dimension you add to the key partitions the cache and lowers the hit rate. A per-principal key on a system with many users may hit almost never, at which point the honest conclusion is that this workload is not cacheable at the response layer — not that the scope should come out of the key.

This is the same class of problem as a retrieval layer that returns content the requester could not otherwise read: authorization applied at the wrong boundary. See the agent threat model for the confused-deputy pattern it belongs to.

The threshold problem

Implementations acknowledge this directly: a semantic cache produces false positives on hits and false negatives on misses. Those two errors are not symmetric, and the asymmetry should drive the threshold.

A false negative shows up as spend. A false positive shows up as a support ticket, weeks later, with no trace explaining it — because from the system’s point of view nothing happened. Tune for precision and accept the lower hit rate.

How to actually set it

Take a few hundred real request pairs from your own traffic and label them: same answer, or different answer. Compute similarity for each, plot the two distributions, and pick a threshold above the overlap rather than through the middle of it. If the distributions overlap heavily, that is the finding — this traffic is not semantically cacheable, and no threshold rescues it.

What never to cache

The first deserves its own sentence because it is a real thing people do. Caching a mutating tool call does not save a duplicate — it suppresses a genuine second action, which is the opposite problem from the one idempotency solves and needs the opposite treatment. If the concern is duplicate effects, the answer is an intent-derived key at the action layer, not a cache at the transport layer; see idempotency for agent actions.

Measuring whether it helped

A semantic cache optimises a metric that improves as correctness degrades. Loosen the threshold and hit rate rises, cost falls, latency falls — every dashboard turns green. The only counterweight is a deliberate audit: sample served hits, compare the stored response against what a fresh call returns, and track the disagreement rate as a first-class metric alongside the savings.

Report the savings and the false-hit rate together or not at all. A 40% hit rate at a 3% false-hit rate is not a 40% saving; it is a trade whose second term someone needs to have agreed to. For where this instrumentation belongs, see agent observability.

The short version: fix the prompt cache first, cache read-only tool results by exact key, and reach for semantic matching only at the edges where requests are genuinely independent — with the authorization scope in the key and a precision-biased threshold you derived from your own traffic.