Evaluating LLM Output Quality: Metrics and Testing
On this page
Shipping a feature backed by a large language model feels deceptively easy. You write a prompt, the demo works, and the temptation is to call it done. But an LLM that produces a brilliant answer in the demo can quietly produce a wrong, unsafe, or off-brand answer in production for inputs you never tried. The difference between a prototype and a dependable product is a rigorous evaluation habit. This post walks through the metrics that actually matter, how to build a testing harness, and how to avoid the traps that make evaluation harder than it should be.
Why LLM Evaluation Is Different
Traditional software has deterministic outputs: given an input, you assert an exact expected value. LLMs break that model in three ways. First, outputs are non-deterministic — the same prompt can yield different text across runs, especially at higher temperatures. Second, there is rarely a single "correct" answer; a good summary can be phrased a thousand ways. Third, quality is multi-dimensional. A response can be factually accurate but verbose, or fluent but subtly wrong.
This means you cannot rely on assertEqual. Instead, you evaluate along dimensions, accept distributions of quality rather than binary pass/fail, and combine automated scoring with human judgment. Think of evaluation as measurement under uncertainty, not verification of correctness.
The Core Quality Dimensions
Before choosing metrics, decide what "good" means for your use case. Most applications care about some subset of these:
- Correctness / factual accuracy — Is the information true and free of hallucination?
- Relevance — Does the response actually address the user's request?
- Completeness — Does it cover everything asked, without important omissions?
- Faithfulness (groundedness) — For RAG systems, does the answer stay anchored to the retrieved context rather than inventing details?
- Coherence and fluency — Is it well-structured and readable?
- Safety — Is it free of harmful, biased, or policy-violating content?
- Format adherence — Does it produce valid JSON, follow the schema, respect length limits?
- Tone and style — Does it match your brand voice?
Weight these by what breaks your product. A legal-document assistant lives and dies on faithfulness; a marketing-copy generator cares more about tone and creativity.
Automated Metrics: What to Use and When
Reference-based metrics
When you have a "gold" reference answer, you can compare against it. Classic metrics like BLEU and ROUGE measure n-gram overlap and are cheap to compute, but they penalize valid paraphrases and correlate weakly with human judgment for open-ended tasks. BERTScore and other embedding-based similarity metrics improve on this by comparing semantic vectors rather than exact tokens, so "the cat sat on the mat" and "a cat was on the mat" score as similar.
Use reference-based metrics for tasks with narrow expected outputs — translation, extraction, or classification — and treat them skeptically for open-ended generation.
Reference-free and functional checks
Many valuable checks need no reference at all:
- Schema/format validation — parse the JSON, validate against a schema, check required fields. This is deterministic and should be the first gate.
- Regex and keyword assertions — confirm required terms appear or forbidden ones don't.
- Executable checks — for code generation, run the code and its tests. This is the gold standard because it's an objective functional signal.
- Retrieval metrics — for RAG, measure context precision and recall to separate retrieval failures from generation failures.
LLM-as-judge
The most flexible modern approach is using a strong model to grade outputs against a rubric. You give the judge model the input, the output, and explicit criteria, and ask for a score with reasoning. This scales far better than human review and handles nuance that string metrics miss.
To make LLM-as-judge reliable:
- Provide a detailed rubric with concrete definitions for each score level, not just "rate 1–5."
- Ask for reasoning before the score so the judgment is grounded.
- Prefer pairwise comparison ("is A or B better?") over absolute scoring when you're comparing two versions — models are more consistent at ranking than rating.
- Calibrate against humans on a sample so you know how far to trust the judge.
- Watch for known biases: judges favor longer responses, their own outputs, and the first option presented. Randomize position and control for length.
Building a Testing Harness
A metric is only useful inside a repeatable process. The backbone of LLM testing is an evaluation dataset: a curated set of representative inputs paired with expected properties or reference answers.
Build it deliberately. Start by mining real production logs for common queries, then add edge cases: ambiguous inputs, adversarial prompts, empty inputs, very long inputs, and examples from each user segment. Aim for coverage of failure modes, not just volume. A focused set of 100–300 well-chosen cases beats 10,000 random ones.
Structure your harness as a pipeline:
- Run each dataset input through your prompt/model/pipeline.
- Score each output with your chosen metrics (format gates first, then semantic and judge-based scores).
- Aggregate into summary numbers — pass rates per dimension, mean scores, and the distribution, not just the average.
- Report with drill-down into individual failures so you can inspect what actually went wrong.
Critically, version everything: the prompt, the model ID and parameters, the dataset, and the results. When quality shifts, you need to know whether the prompt changed, the model was updated, or the input mix drifted.
Regression Testing and CI Integration
Treat evaluation like a test suite. When you change a prompt or upgrade a model, re-run the harness and compare against the previous baseline. Set thresholds — for example, "faithfulness pass rate must stay above 95%" — and fail the build if a change regresses below them.
Because runs are non-deterministic and API calls cost money and time, don't run the full 300-case suite on every commit. A practical tiering:
- A small smoke set (10–20 cases) on every pull request for fast signal.
- The full suite nightly or before releases.
- Targeted sets when touching a specific feature.
Pin temperature to 0 for tests where you want maximum reproducibility, and run multiple samples per input where you need to measure variance rather than eliminate it.
Common Pitfalls
- Optimizing to the metric. If you tune only to raise a ROUGE score, you may degrade real quality. Metrics are proxies; keep a human in the loop periodically.
- Tiny or stale datasets. A dataset that never grows stops reflecting production. Continuously fold in new failure cases.
- Averaging away failures. A strong mean score can mask a meaningful rate of catastrophic errors in the tail. Always look at the distribution, not just the average.
- Trusting the judge blindly. An uncalibrated LLM judge can be confidently wrong. Sample and verify against humans.
- Ignoring cost and latency. Quality isn't the only axis. Track tokens, dollars, and response time alongside quality so you see the full trade-off.
Putting It Together
A mature evaluation setup layers defenses: deterministic format gates catch structural failures instantly, semantic and functional checks measure substance, LLM-as-judge scores nuance at scale, and periodic human review keeps everything honest. All of it runs against a versioned, growing dataset inside a harness wired into your release process. Start small — a dozen cases and one judge prompt — and expand as you discover new ways your system fails.
FAQ
How large should my evaluation dataset be? Start with 50–100 carefully chosen cases covering your main use cases and known edge cases. Quality and coverage matter far more than size. Grow it over time by adding real failures you discover in production.
Are BLEU and ROUGE still worth using? They're useful for narrow tasks with predictable outputs, like translation or extraction, and they're cheap. For open-ended generation, they correlate poorly with human judgment — prefer embedding-based similarity or LLM-as-judge there.
Can I trust an LLM to grade another LLM? Yes, with care. LLM-as-judge is reliable when you give it a clear rubric, ask for reasoning, control for known biases (length, position, self-preference), and calibrate against human ratings on a sample. Treat it as a strong signal, not ground truth.
How do I handle non-deterministic outputs in tests? Set temperature to 0 for reproducible checks, assert on properties rather than exact strings, and where variance itself matters, sample each input several times and measure the distribution instead of expecting one answer.
What's the single most important thing to measure? It depends on your product's failure mode. For RAG and factual assistants, faithfulness/groundedness is usually the highest-stakes dimension. Identify what causes the worst real-world harm for your users and weight that dimension most heavily.
How often should I run evaluations? Run a small smoke set on every change for fast feedback, the full suite nightly or before releases, and always re-evaluate when you change prompts, upgrade models, or adjust parameters.