Toxicity and Hallucination Filtering
Use layered checks to reduce harmful, fabricated, or unsupported answers.
Separate the problems
Toxicity concerns harmful, abusive, or demeaning content, while hallucination concerns claims that are fabricated, unsupported, or presented with unjustified confidence. They need different signals and mitigations. A response can be polite but false, or factual but unsafe in context. Define the failure categories separately so evaluation and product controls do not collapse them into one vague “bad output” label.
Input and output moderation
Classifiers, policy rules, blocklists, and human escalation can identify requests or responses that need refusal, transformation, or review. Consider context, reclaimed language, quotation, and educational use to reduce simplistic false positives. Moderation scores are signals, not final authorization. Apply checks to streamed and final output, and decide what happens when a classifier is unavailable.
def answer_or_review(answer, citations, moderation):
if moderation != "pass" or not citations:
return {"status": "needs_review", "answer": None}
return {"status": "supported", "answer": answer, "citations": citations}
print(answer_or_review("Claim", ["source-1"], "pass"))Grounding answers
Reduce hallucination by supplying relevant evidence, requiring citations, validating extracted fields, and instructing the model to abstain when sources do not support a claim. Retrieval quality matters: irrelevant or stale context can make an answer look grounded while still being wrong. Compare each important claim with the source where feasible, especially for legal, financial, medical, or operational decisions.
Structured and deterministic checks
Schemas can catch missing fields, invalid values, and unsupported categories, while business rules can check arithmetic, dates, permissions, and allowed actions. These checks are stronger than asking the model to “be accurate.” They should fail visibly and route to correction or review. Do not silently coerce an unsafe value into a valid-looking one.
Uncertainty and abstention
A reliable assistant knows when evidence is weak or the request is outside scope. Design response states such as supported answer, clarification needed, insufficient evidence, and human review. Avoid forcing every request into a confident paragraph. Measure whether uncertainty language correlates with actual errors, and improve the underlying workflow when the model is uncertain for predictable reasons.
Measuring filter performance
Track true and false positives, unsafe completions, unsupported claims, refusal quality, user reports, and reviewer overrides. Test adversarial phrasing, multilingual content, sarcasm, quotations, long context, and attempts to split a harmful request across turns. Review examples regularly because input distributions and model behavior change. An average safety score can hide a rare but severe failure.
Layered response design
Use moderation, grounding, validation, permissions, human review, and incident response together. Give users a safe correction path and a way to report harmful or incorrect content. Preserve enough redacted evidence to investigate. Filters reduce risk, but they do not make a model authoritative; the product must still set boundaries and keep people accountable for consequential outcomes.