What is LLM observability?
LLM observability is the practice of understanding what a language model application actually did at runtime. Ordinary observability tracks requests, errors, and latency. LLM observability adds the parts that are unique to language models: the full prompt sent to the model, the response it returned, the tokens consumed, the cost of the call, and a judgement of whether the answer was correct and useful.
It matters because LLM apps fail differently from normal software. A function either returns the right value or throws an error. A language model can return a fluent, confident, well-formatted answer that is simply wrong. Nothing crashes, no exception fires, and without the right visibility you never notice.
In plain words
Normal monitoring is like checking that a delivery arrived on time. LLM observability is opening the box to check that what is inside is actually what you ordered. The package can show up fast, look perfect, and still contain the wrong thing.
What you watch
- Prompts and responses. The exact input and output of every call. When an answer is bad, this is the first thing you read.
- Traces across steps. Real apps chain many calls: retrieval, a tool call, then generation. A trace shows the whole chain so you can see which step went wrong.
- Tokens and cost. Each call has a price. Without tracking it, a single inefficient prompt can quietly run up a large bill.
- Latency. Models can be slow. You watch response time the way you would for any user-facing service.
- Quality. Through user feedback, automated checks, or a second model scoring the output, you measure whether answers are actually good, not just whether they were produced.
Why it matters
- Hallucinations are invisible without it. A wrong answer looks identical to a right one in the logs unless you capture and evaluate the content.
- Cost can spiral fast. Token-based pricing means a bad prompt or a retry loop turns into real money. You catch it only if you measure it.
- Debugging needs the full chain. "The answer was wrong" is useless on its own. Seeing the retrieved context and every prompt tells you whether retrieval, the prompt, or the model failed.
Common pitfalls
- Logging only inputs and outputs. Without the retrieved context, the chosen model, and the parameters, you cannot reproduce or explain a bad answer.
- No quality signal at all. If you track cost and latency but never measure whether answers are correct, you are watching the cheap, easy half and missing the point.
- Capturing sensitive data carelessly. Prompts and responses often contain personal or confidential information. Decide upfront what you store, mask, and retain.
- Evaluating only in testing. Real user inputs are messier than your test set. Keep measuring quality in production, not just before launch.
Related articles:
- What is observability? - The broader practice of understanding running systems through the data they emit.
- What is an LLM? - The language model whose behaviour you are trying to see into.
- What is Retrieval-Augmented Generation (RAG)? - The pattern whose retrieval step is a common source of bad answers worth tracing.
Want to stay one step ahead?
Don't miss our best insights. No spam, just practical analyses, invitations to exclusive events, and podcast summaries delivered straight to your inbox.