Skip to content

Monitoring LLM Applications

What to track once an LLM application is live: quality, safety, cost, latency and user feedback.

Editorial team 1 min read

LLM applications need monitoring beyond standard application metrics.

Operational Metrics

  • Latency, including time to first token.
  • Error rates and timeouts.
  • Token usage and cost per request, user and feature.
  • Rate limit hits.

Quality Signals

  • User feedback: ratings, edits, regenerations.
  • Automated evaluation on sampled traffic using rubrics or model graders.
  • Retrieval quality for RAG: empty results, low relevance.
  • Task completion rates.

Safety Signals

  • Safety classifier flags.
  • Refusal rates, both too high and too low.
  • Prompt injection attempts.
  • Personal data in outputs.

Tracing

Record prompts, retrieved context, tool calls and outputs for debugging, with redaction and retention limits.

Drift

User behaviour and topics change; provider models get updated. Watch for shifts in metrics.

Acting on Data

Review samples regularly, turn failures into test cases, and alert on sudden changes.

More in MLOps & deployment

All MLOps & deployment guides →
MLOps & deployment Guide · 2 min

What Is MLOps?

The practices that take machine learning from notebook to reliable production: versioning, automation, deployment and monitoring.

MLOps & deployment 2 min read 26 Dec 2025

MLOps & deployment Guide · 2 min

Deploying Machine Learning Models

Batch scoring, real-time APIs, streaming and on-device inference: choosing how predictions reach users, and deploying safely.

MLOps & deployment 2 min read 25 Dec 2025

MLOps & deployment Guide · 2 min

Monitoring Machine Learning Models in Production

What to monitor after deployment — data, predictions, outcomes and operations — and how to respond when things change.

MLOps & deployment 2 min read 24 Dec 2025

MLOps & deployment Guide · 2 min

Model Registries and Versioning

Why every production model needs a version, metadata and lineage, and how a model registry manages promotion and rollback.

MLOps & deployment 2 min read 23 Dec 2025