LLM Observability: What to Measure for Self-Hosted Models
LLM observability means being able to answer, from data you already collect, why your model service is slow, failing or expensive right now. For a self-hosted model that comes down to a short list: request volume, latency (including time to first token and tokens per second), token usage per team and app, error rates, GPU utilization and memory, and queue depth.
With a hosted API, the provider watches the GPUs and you watch your bill. When you run the model yourself, the GPUs are yours too, so the metrics have to cover both the application side and the hardware side.
This guide lists what to measure, what to log about prompts without creating a privacy problem, sensible starting alerts, the tools most teams use, and the dashboards to build first.
LLM observability metrics for self-hosted inference
Requests and errors
Count requests per model, per app and per status code. Split errors by cause, because each one has a different fix:
- Context length exceeded: a client is sending more tokens than the server allows. Fix the client or raise the limit.
- Timeouts: the server is overloaded or a request is generating far too many tokens.
- Out of memory: the KV cache (the memory that holds each conversation's state) ran out. Too many concurrent long requests for the GPU.
- Rejected by auth: an expired or missing API token, usually a deploy problem on the client side.
Latency: time to first token and tokens per second
One latency number hides the two things users feel. Time to first token (TTFT) is how long the user waits before anything appears. It grows with prompt length and with queueing. Tokens per second (output speed) decides how long the full answer takes to stream.
Total time for a response is roughly:
total = TTFT + output tokens / tokens per second
A 500-token answer at 40 tokens per second with 0.8 s TTFT takes 0.8 + 500 / 40 = 13.3 s. If tokens per second drops to 20, the same answer takes 25.8 s. Track both as percentiles (p50, p95, p99), not averages: an average of 2 s can hide a p99 of 30 s.
Token usage per team and app
Input and output tokens per request, tagged with the model, the app and the team that sent it. This is your cost signal. On your own hardware a token has no price tag, but tokens translate into GPU time, and GPU time is what you budget. Token totals by team are also how you answer "who is using the 70B model and for what".
GPU utilization and memory
GPU utilization tells you whether you are paying for idle hardware or running out of headroom. GPU memory tells you how close you are to out-of-memory errors. Watch memory used against the total, and if your inference server reports it, KV cache usage as a percentage. Our guide to GPU memory for LLM inference explains why the KV cache, not the weights, is usually what fills up.
Queue depth
Requests waiting for a slot on the server. This is the earliest warning of overload, because the queue grows before latency and errors do. You can predict it with Little's law: requests in flight = arrival rate x average duration.
At 2 requests per second and 13 s per request, 2 x 13 = 26 requests are in flight on average. If the server processes 16 at once, about 10 are always waiting, and TTFT climbs for everyone. Either add capacity or reduce the average duration (shorter outputs, a smaller model for simple work).
What to log about prompts
Prompts and responses are the most useful debugging data you have and the most sensitive. They contain whatever users paste in: customer records, source code, contracts. Self-hosting keeps that data inside your environment, but a log store full of prompts is still a new copy of sensitive data that needs protecting.
A sensible default:
- Always log metadata: timestamp, model, app, team, user or token ID, input and output token counts, latency, TTFT, status and error type. No content.
- Log content only when there is a reason: a debugging window for one app, an evaluation set built with consent, or a sampled share of traffic with redaction applied first.
- Store content separately from metrics, with tighter access control and a short retention period, such as 7 to 30 days.
- Hash prompts if you need to spot repeated requests without storing them.
The OpenTelemetry GenAI conventions take the same line: the attributes that carry message content are opt-in, and the span specification warns they are likely to contain sensitive information including personal data. If you work under privacy rules, your legal and security teams decide retention. This post is not legal advice.
Alert thresholds to start with
There are no standard numbers. Measure a week of normal traffic, then alert on departures from that baseline. These starting points work for most internal deployments:
| Signal | Starting alert | Why |
|---|---|---|
| Error rate | > 1% of requests over 5 minutes | Normal traffic should be well under this |
| p95 TTFT | > 2x the weekly baseline for 10 minutes | Catches queueing before users complain |
| Tokens per second (p50) | < 50% of baseline for 10 minutes | Points at GPU contention or a bad deploy |
| Queue depth | > 0 sustained for 5 minutes | A queue that never drains means you are over capacity |
| GPU memory | > 95% for 10 minutes | Out-of-memory errors are close |
| Requests | Zero for 15 minutes in working hours | A silent outage looks like a quiet day |
Page someone only for errors and the zero-traffic alert. The rest can go to a channel and be reviewed the same day. Too many pages and people stop reading them.
Tooling options
Prometheus and Grafana
Prometheus is an open-source monitoring and alerting toolkit that stores metrics as time series identified by a name and labels, and pulls (scrapes) them over HTTP. Grafana includes Prometheus as a built-in data source for dashboards. Together they are the default stack for self-hosted metrics, and they run fine without internet access.
Many inference servers expose Prometheus metrics directly. vLLM's metrics at /metrics include vllm:time_to_first_token_seconds, vllm:num_requests_waiting (queue depth) and vllm:kv_cache_usage_perc. The llama.cpp server has a Prometheus-compatible endpoint you turn on with --metrics. For the GPUs themselves, NVIDIA's DCGM Exporter publishes GPU telemetry in Prometheus format.
OpenTelemetry GenAI semantic conventions
OpenTelemetry is the open standard for traces, metrics and logs. Its GenAI semantic conventions define common names for model calls, so data from different tools lines up. The GenAI metrics include gen_ai.server.time_to_first_token, gen_ai.server.time_per_output_token and gen_ai.server.request.duration. These conventions are marked as in development and may still change, so pin the version your tools use and check the current release.
Use OpenTelemetry when one user action fans out into several model calls (a RAG pipeline or an agent), because a trace shows each step's latency in one view. For a single chat endpoint, Prometheus metrics are usually enough.
Dashboards to build first
Build three, in this order.
- Service health. Requests per minute, error rate by type, p50/p95 TTFT and tokens per second, per model. This is the one on the wall screen.
- Capacity. GPU utilization, GPU memory and KV cache usage, queue depth and in-flight requests, per server. This is the one you open when health turns red, and the one you use to plan hardware with self-hosted LLM cost in mind.
- Usage. Input and output tokens by team, app and model over 30 days. This answers chargeback questions and shows which apps could move to a smaller model through LLM routing.
Resist building more until these three have been used during a real incident. Each panel should answer a question someone actually asked.
Where AI Refinery fits
AI Refinery, a self-hosted LLM platform, includes observability for the models you deploy with it: real-time monitoring of every inference server, covering request volume, latency, token usage and errors, in one dashboard. The Basic plan has basic observability, Professional has full observability, and Enterprise can be customized. Alongside it, the audit log records who used which model, which supports the governance side described in our LLM governance guide.
It is not a complete observability stack on its own. If you need GPU-level telemetry, traces across a multi-step agent or RAG pipeline, or long-term metrics in the Prometheus and Grafana setup your platform team already runs, keep those tools and run them alongside it. And if you only have one model and a handful of users, a server's built-in metrics endpoint and one Grafana dashboard may be all you need for now.