Agent observability: Langfuse vs Arize Phoenix vs Braintrust

A note on method.This is a capability mapping drawn from each vendor’s current documentation, not a hands-on bake-off: we have not run the same instrumented agent through all three and measured them. Where that matters — and for anything that changes quickly, like pricing and deployment options — verify against the primary sources listed at the end. What follows that is durable: the trace requirement and the architectural decision, which outlive any of these products.

What agent observability means

That distinction is the whole reason a dedicated tool exists. A conventional APM will happily tell you that a request took 14 seconds and returned 200. For an agent, neither number is the interesting part. The interesting part is that on step four it read a document it should not have had access to, and on step five it acted on what it found there.

The minimum trace

Work out this list before you shortlist anything. It is what you are actually buying, and it is the only basis on which a feature matrix means anything.

Rows two through four come nearly free from any of these tools once instrumented. Rows one and seven need a little work. Rows five and six need real work, and nobody sells them — which is worth knowing before you assume a purchase has covered your audit requirement. If your governance layer decides what an agent may do, then the record of which rule fired is part of the evidence, and it has to be emitted deliberately.

Token accounting is not a nice-to-have

Row two’s token fields deserve emphasis because they are the fastest way to find the expensive bug. Cached versus uncached input tokens tell you whether your prompt cache is working at all — and a cache-read count of zero across repeated, apparently-identical requests is the signature of a silent invalidator, usually a timestamp or an ID interpolated into the system prompt. Without that field in the trace, you are guessing.

The question that actually decides it

This is a platform decision, not a features decision, and answering it first eliminates candidates faster than reading any comparison table. Both answers are defensible.

The portability row is the one people underweight and later regret. The instrumentation is the expensive, invasive part of adoption; the dashboard is not. A pipeline you can point at a different backend is worth more than any single feature you are comparing today, because this product category is about eighteen months old and will not look like this in two years.

The three tools

Langfuse

Open source and self-hostable, and explicitly positioned as purpose-built for LLM applications rather than a general APM — with native handling of LLM concepts like token usage and model parameters. Traces capture the full request lifecycle including model calls, retrieval, tool executions and custom logic, organised as traces, sessions and observations.

Beyond tracing it documents LLM-as-a-judge evaluation, datasets and experiments, prompt management with versioning, and custom dashboards. The SDKs send data asynchronously in the background, which matters more than it sounds: instrumentation that blocks your agent loop is instrumentation you will eventually remove.

Arize Phoenix

Built by Arize AI with the open-source community, and architecturally the most conventional of the three: it accepts traces over OTLP and uses OpenInference as its instrumentation framework. That gives it broad auto-instrumentation — documented coverage includes LlamaIndex, LangChain, DSPy, Mastra and the Vercel AI SDK, plus model providers including OpenAI, Bedrock and Anthropic, across Python, TypeScript and Java.

Feature coverage is comparable to Langfuse: tracing of model calls, retrieval and tool use; evaluation via LLM-based evaluators, code-based checks or human annotation; prompt versioning with a playground and span replay; and datasets and experiments for comparing versions. Deployment runs local, self-hosted via Docker or Kubernetes, or cloud.

Span replay is the underrated item on that list. Re-running a single historical span against a changed prompt is the tightest possible iteration loop for a specific failure, and it is much faster than reproducing the whole trajectory.

Braintrust

Positions itself as an active observability platform for instrumenting, understanding and improving agents, with the emphasis on applying intelligence to traces and surfacing significant patterns automatically rather than leaving you to find them. Documented surface includes tracing, evaluations, playgrounds, datasets, a bt CLI for scriptable instrumentation, and an MCP server for IDE integration. The documented workflow runs instrument → observe → annotate → evaluate → deploy.

Two things to establish yourself before shortlisting it: its docs present a cloud offering without advertising the open-source self-hosted path that Langfuse and Phoenix both do, and open-source status is not stated. If either matters to you, confirm directly — those are hard constraints, and hard constraints should do the filtering before features get a vote.

Choosing

On the first row: prompts and outputs are among the most sensitive data your system handles, because they contain whatever your users and your documents contain. A tool that requires sending them to a third party is a data-processing decision, not a tooling decision, and it belongs with whoever owns that call.

On the last row, check auto-instrumentation against the framework you actually use, not the length of the supported list. Coverage for a framework you do not use is worth nothing, and hand-instrumenting an unsupported one is the single largest adoption cost in this category.

What none of them do for you

Three gaps are consistent across the category.

The authority record. Rows five and six of the minimum trace. Your governance layer knows which rule fired and who approved what; the observability tool only knows a call happened. Emitting that as a span attribute is a few lines of work and it is the difference between reconstructing an incident and speculating about one.

The definition of good. Every one of these ships LLM-as-a-judge evaluators, and none of them can tell you what your acceptance criteria are. That work is yours, it is domain-specific, and it is the part that actually determines whether the eval suite is worth running.

Retention that matches your obligation. Default retention is set for debugging — weeks. If a trace is your evidence that an agent acted within its authority, the retention requirement comes from your compliance posture, not from a pricing tier. Check this before you need it.

Instrument before you choose

// The point is not the abstraction — it is that the call sites,
// which are scattered through your agent, never name a vendor.
interface AgentTracer {
  run(objective: string): Span;
  modelCall(span: Span, d: { model: string; tokensIn: number; tokensOut: number; cachedIn: number }): void;
  toolCall(span: Span, d: { tool: string; args: unknown; result: unknown; isError: boolean }): void;

  // The two nobody provides — emit them yourself.
  permissionDecision(span: Span, d: { tool: string; decision: "allow" | "deny" | "ask"; rule: string; step: string }): void;
  approval(span: Span, d: { actor: string; decidedAt: string; shown: string }): void;

  finish(span: Span, outcome: "completed" | "exhausted" | "escalated" | "failed"): void;
}

The two highlighted methods are the ones this article exists to argue for. They cost almost nothing to add at the start, they cannot be backfilled, and they are what makes the trace usable by someone other than the engineer debugging it.

For where this layer sits relative to everything else — and why building it before the governance layer rather than after is the right order — see the seven-layer stack. For what the permission decisions you are recording actually look like, see the Agent SDK’s six-step evaluation order.