Designing Metrics: Exact Checks, Rubrics, and Reference Answers
Learn how to choose the right kind of check for different types of AI output, from deterministic validation to human rubrics.
Match the metric to the task shape
A task with one correct answer (a extracted field, a classification label, valid JSON matching a schema) should be graded with an exact, deterministic check — it's cheap, reproducible, and unambiguous. A task with many acceptable answers (a summary, an explanation, a creative response) needs a rubric or comparative judgment instead, because exact match would reject perfectly good answers that happen to be worded differently.
Deterministic checks first
Before reaching for a rubric or a judge, ask whether part of the output can be checked programmatically: does the JSON validate against the schema, does the cited page number exist in the source document, does the code compile and pass a test. Deterministic checks are the cheapest, most trustworthy signal available and should carry as much of the evaluation as the task allows.
Writing a rubric that produces consistent scores
A usable rubric defines specific, observable criteria — 'cites at least one source for every factual claim,' 'does not contradict the provided context,' 'stays within the requested length' — rather than a single vague 'quality' scale. Give the rubric to two different reviewers (or the same reviewer twice) on the same examples and check agreement; a rubric that produces wildly different scores from different graders needs tightening before it's trusted.
Reference-based comparison
For tasks with a known good answer, compare the model's output against a reference using either exact match (for short, constrained answers) or a similarity or entailment check (for longer text, where wording will legitimately vary). Reference-based grading is only as good as the reference set, so invest in writing or curating high-quality references rather than assuming any prior answer is a valid one.
Composite scoring and what to optimize
Real features usually need more than one metric — correctness, safety, latency, cost — and a single blended score can hide a critical failure behind an average. Report each dimension separately, and set hard gates on the dimensions that must never regress (a safety violation, an invalid schema) rather than letting them be averaged away by strength elsewhere.
Practical exercise
Take the eval set from the previous lesson and, for each example, decide whether it needs a deterministic check, a rubric, or reference comparison. Write the specific rubric criteria for at least one non-deterministic example, and have someone else score three outputs against it. Compare your scores and refine the rubric where you disagreed.
Sources and further reading
These primary or specialist references informed the concepts in this guide. Product details can change, so verify current documentation before implementation.