How to Evaluate and Monitor LLM Quality in Production

Traditional software is deterministic. Given the same input, you get the same output. You write unit tests, they pass or fail, and you ship with confidence. LLMs break this model completely.

The same prompt can produce different outputs on consecutive calls. "Good" output is subjective and context-dependent. A response that's perfect for a casual chatbot might be inappropriate for a legal document generator. And the model itself changes when providers push updates, sometimes without notice.

Most teams skip evaluation entirely and rely on vibes. They run a few manual tests, the outputs look reasonable, and they deploy. Then users start complaining about hallucinations, inconsistent tone, or wrong answers, and nobody has the data to diagnose what changed or why.

The companies that build reliable AI features treat evaluation as a core part of their engineering process, not an afterthought. They have automated test suites, production monitoring, and quality dashboards just like they have for any other critical system.

A robust evaluation system operates at three levels, each catching different types of quality issues:

Layer 1: Offline Evaluation (Before Deployment)

Run your LLM against a curated test dataset before any code reaches production. This is your gate check. If accuracy drops below your threshold, the deployment doesn't happen. Build a test suite of 100 to 500 examples covering your most important use cases, edge cases, and known failure modes. Run this suite on every prompt change, model upgrade, or configuration update.

Layer 2: Online Monitoring (During Production)

Track quality metrics on every production request in real time. This catches issues that offline evaluation misses: distribution shifts in user inputs, model provider degradation, and edge cases you didn't anticipate. Sample 5% to 20% of production traffic for detailed evaluation, and monitor 100% for basic metrics (latency, error rates, output length).

Layer 3: Human Review (Continuous Improvement)

Regularly have humans review a sample of production outputs. This calibrates your automated metrics, identifies new failure patterns, and builds training data for improving your evaluation models. Weekly review sessions of 50 to 100 outputs keep your team connected to actual quality.

Each layer has different cost and coverage characteristics. Offline evaluation is cheap but limited to known scenarios. Online monitoring is comprehensive but requires infrastructure investment. Human review is high-quality but expensive and slow. You need all three.

Your test dataset is the foundation of everything else. A bad test dataset makes all your evaluation metrics meaningless. Here's how to build a good one:

Sources for Test Examples

• Production logs. Sample real user inputs from your application. These represent actual usage patterns, not hypothetical scenarios. Anonymize personal data before using them.
• Support tickets and bug reports. Every user complaint about AI quality is a test case. If a user reported a hallucination, that input/output pair belongs in your test suite.
• Edge cases. Deliberately craft inputs that test boundaries: very long inputs, ambiguous questions, questions outside your domain, adversarial prompts, multilingual inputs.
• Golden examples. Have domain experts write ideal responses for a subset of inputs. These "gold standard" examples serve as reference points for automated scoring.

Test Dataset Structure

Each test case should include: the input (user message plus any context), the expected output or acceptable output criteria, the category (so you can track accuracy per use case), and difficulty level (easy, medium, hard). Tag each example so you can slice evaluation results by category and identify specific weak spots.

Maintaining the Dataset

Your test dataset is a living document. Add new examples monthly from production failures. Remove examples that no longer represent real usage patterns. Revalidate golden examples quarterly to ensure they still reflect your quality standards. A stale test dataset gives you false confidence.

Different use cases need different metrics. Here are the ones that actually matter:

Factual Accuracy

Does the output contain correct information? For RAG applications, check whether the response is grounded in the retrieved documents. For data queries, compare the answer against a known-correct result. Automated approaches: use a separate LLM call to verify factual claims against source documents, or compare structured outputs against expected values.

Relevance

Does the output actually answer the question? A factually correct response that doesn't address the user's question is still a failure. Measure with embedding similarity between the question and the answer, or use an LLM judge to score relevance on a 1 to 5 scale.

Hallucination Rate

How often does the model state things that aren't supported by the provided context? This is critical for any application where users trust the AI's output. Detect hallucinations by checking whether each claim in the output can be traced back to a source document or known fact.

Consistency

Does the model give similar answers to similar questions? Run the same prompt 5 times and measure output variance. For classification tasks, expect near-zero variance. For generation tasks, the core information should be consistent even if the wording varies.

Format Compliance

Does the output follow the required format? If you asked for JSON, is it valid JSON? If you asked for a bullet list, did you get one? This is the easiest metric to automate and often the first thing to break when prompts or models change.

Toxicity and Safety

Does the output contain harmful, biased, or inappropriate content? Use classifiers like Perspective API or dedicated content safety models to flag outputs that violate your guidelines. Set a zero-tolerance threshold for production applications.

The most powerful evaluation technique in 2026 is using one LLM to evaluate another's output. It's faster and cheaper than human review, and correlates well with human judgment when done correctly.

How It Works

You send a separate LLM (the "judge") the original input, the generated output, and a scoring rubric. The judge returns a score and explanation. For example: "Rate this customer support response on a scale of 1 to 5 for helpfulness, accuracy, and tone. Explain your reasoning."

Best Practices

• Use a different model as the judge. Don't use Claude to evaluate Claude or GPT-4 to evaluate GPT-4. Models tend to rate their own outputs more favorably. Use Claude as a judge for GPT-4 outputs and vice versa.
• Provide detailed rubrics. Vague instructions produce vague scores. Define exactly what a 1, 3, and 5 look like for each dimension. Include examples of good and bad outputs at each score level.
• Use pairwise comparison. Instead of absolute scoring, show the judge two outputs and ask which is better. This produces more reliable rankings than independent scores.
• Calibrate against human judgment. Regularly compare LLM judge scores against human evaluator scores. If they diverge, update your rubric. Expect 80% to 90% agreement between a well-calibrated LLM judge and human evaluators.

Cost of LLM-as-Judge

Each evaluation call costs roughly the same as a regular API call. If you're evaluating 1,000 outputs per day using Claude Sonnet, expect $5 to $15/day in evaluation costs. That's a tiny fraction of the value you get from quality visibility.

You need tooling to run evaluations, store results, and alert on quality drops. Here are the options:

LangSmith (LangChain)

The most comprehensive LLM observability platform. Traces every LLM call with full input/output logging, latency tracking, and token usage. Built-in support for running evaluations against test datasets and comparing runs. Pricing starts free for small teams, $39/month for production use. Our default recommendation for teams using LangChain.

Braintrust

Purpose-built for LLM evaluation. Excellent UI for reviewing outputs, running A/B tests between prompts, and tracking metrics over time. Strong support for LLM-as-judge evaluation with customizable scorers. Good for teams that want a dedicated evaluation platform separate from their orchestration framework.

Helicone

Lightweight proxy that sits between your application and the LLM API. Logs every request with minimal code changes (swap the base URL). Provides cost tracking, latency monitoring, and basic quality dashboards. Great for teams that want observability without a full platform commitment. Free tier covers most startups.

Custom Pipeline

For teams with specific requirements, build a custom evaluation pipeline: log outputs to a database (PostgreSQL or BigQuery), run evaluation scripts on a schedule (hourly or daily), store scores alongside the original outputs, and build dashboards in Grafana or Metabase. More work upfront but maximum flexibility. Development cost: $5,000 to $15,000.

Alerting

Set up alerts for quality regressions. If your average accuracy score drops by more than 10% over a rolling 24-hour window, someone should get paged. If hallucination rate exceeds your threshold, pause the feature and investigate. Treat LLM quality alerts with the same urgency as application error rate alerts.

Evaluation isn't a one-time setup. It's an ongoing process that makes your AI features better over time:

Weekly Quality Reviews

Set aside 1 to 2 hours per week to review a sample of production outputs. Look for patterns in failures. Are certain types of questions consistently handled poorly? Are there new user behaviors your system wasn't designed for? Turn every failure pattern into a test case and a prompt improvement.

Prompt Iteration Loop

When you identify a quality issue, update your prompt, run your offline test suite, compare scores against the previous version, and deploy only if the new prompt improves the target metric without regressing others. This scientific approach to prompt engineering is the difference between reliable AI features and fragile ones.

Model Upgrade Testing

When a provider releases a new model version, don't switch immediately. Run your full evaluation suite against the new model. Compare accuracy, latency, and cost against your current model. Some "upgrades" perform worse on specific tasks. Data-driven model selection beats hype-driven model selection every time.

The Feedback Flywheel

Production failures become test cases. Test cases drive prompt improvements. Improved prompts produce better outputs. Better outputs generate fewer user complaints. Fewer complaints mean more user trust. More trust means more usage. More usage produces more production data. More data enables better evaluation. This flywheel is how AI features go from "pretty good" to "indispensable."

The teams that win with AI aren't the ones that build the fanciest features. They're the ones that build the most rigorous evaluation systems. Quality compounds. If you're serious about shipping reliable AI features, invest in evaluation infrastructure from day one.

Need help building an LLM evaluation pipeline for your product? Book a free strategy call and we'll help you design a quality monitoring system that scales with your AI features.