Evaluating LLM Outputs
Build practical evaluations for correctness, relevance, safety, and consistency.
Define success before grading
An evaluation needs a clear task, input distribution, expected properties, and decision threshold. “Sounds good” is not a stable requirement. Specify whether the output must be factually correct, grounded in sources, complete, concise, polite, schema-valid, or safe to refuse. Different tasks require different rubrics, and one score should not hide a failure in a critical dimension.
Build representative cases
Collect ordinary requests, edge cases, ambiguous requests, adversarial inputs, and examples that caused real user corrections. Remove or replace private data while preserving the difficulty. Separate development, validation, and held-out sets. Include multilingual and long-context examples when the product supports them. A small but representative set is more useful than a large collection of nearly identical prompts.
def passes(result):
return (
result["schema_valid"]
and result["grounded"]
and result["safety_review"] == "pass"
)
print(passes({"schema_valid": True, "grounded": True, "safety_review": "pass"}))Deterministic checks
Use code for properties that code can judge: valid JSON, required fields, allowed values, citation identifiers, numeric ranges, prohibited secrets, and latency limits. Deterministic checks are fast and repeatable, so they should run on every candidate release. They cannot assess every nuance, but they prevent obvious regressions from being hidden by a favorable subjective score.
Reference and rubric grading
Reference answers help when there is a known factual target, while rubrics work better for open-ended quality. State what earns full, partial, and failing credit with concrete examples. Model-assisted graders can scale review but need calibration against humans and should not be trusted for high-impact decisions without oversight. Preserve grader prompts and versions as evaluation dependencies.
Safety and abstention
A good answer is not always an answer. Test whether the system refuses unsafe requests, asks for missing information, acknowledges uncertainty, and avoids inventing evidence. Include indirect instructions in retrieved documents and attempts to extract secrets. Measure false refusals as well as unsafe completions, because an over-aggressive system can block legitimate work and encourage users to bypass controls.
Regression analysis
When a candidate loses, inspect the examples rather than only reading the aggregate score. Group failures by retrieval, reasoning, formatting, language, safety, and product logic. A model change may improve average quality while damaging one important segment. Keep a failure corpus and turn confirmed regressions into permanent tests so the same lesson is not forgotten.
Using evaluations in release decisions
Set thresholds for must-pass checks and acceptable movement for softer metrics. Compare quality with cost, latency, and operational reliability under production-like settings. Document known weaknesses and the mitigation that covers them. Evaluation is a decision instrument: it should tell the team whether to ship, revise, limit exposure, or add human review—not merely produce a leaderboard.