Preparing Datasets
Build clean, representative examples that make fine-tuning more likely to work.
Define the target behavior
Start with a written task contract: accepted inputs, desired outputs, forbidden behavior, quality thresholds, and escalation conditions. Decide whether examples are conversations, classifications, extractions, rankings, or transformations. A vague goal produces a dataset that teaches multiple incompatible behaviors. Collect examples from realistic workflows, but remove information that is not needed. Dataset design should make it obvious what the model is expected to learn and what must remain a runtime rule or retrieved fact.
Clean and normalize
Remove exact duplicates, near duplicates, malformed records, accidental secrets, unnecessary personal data, and examples with unresolved labels. Normalize role names, field types, whitespace, encodings, and date conventions without erasing meaningful variation. Preserve a raw source outside the training export when governance permits, and record every transformation. Run schema checks that fail loudly on missing inputs, empty targets, invalid JSON, overlong sequences, and unrecognized labels. Silent repair makes debugging and audit much harder.
import json
def validate_record(line: str) -> None:
record = json.loads(line)
assert isinstance(record.get("input"), str) and record["input"].strip()
assert isinstance(record.get("output"), str) and record["output"].strip()
assert len(record["input"]) <= 8_000
assert len(record["output"]) <= 4_000Quality and annotation
Write annotation guidance with positive examples, counterexamples, refusal criteria, and rules for ambiguity. Measure agreement on a sample and adjudicate disagreements before scaling collection. Review labels for factual correctness, tone, policy compliance, and hidden assumptions. If specialists are needed, document who is qualified and how their decisions are checked. Human-generated data is not automatically high quality; fatigue, inconsistent interpretation, and copied errors can become persistent model behavior. Revisit guidance after each review round.
Coverage and balance
Include common requests, rare but important edge cases, incomplete inputs, contradictory context, spelling variation, realistic lengths, multilingual examples where relevant, and safe refusals. Balance classes and outcomes so the model does not learn that the majority answer is always appropriate. Avoid oversampling one template until the model memorizes superficial wording. Use a coverage matrix to connect each requirement and failure mode to examples, and mark gaps that still need collection rather than hiding them.
Prevent leakage
Training data should not contain evaluation answers, future production labels, duplicated customer cases across splits, or metadata that reveals the target directly. Inspect examples for personal identifiers, credentials, proprietary documents, and copyrighted material according to your policy. Deduplicate by normalized text and meaningful entities, not just row IDs. If the product must delete a person’s data, know whether it entered a training artifact and what retraining or exclusion process is required. Data lineage is a production concern.
Split and version
Create train, validation, and test sets before tuning, using a split strategy that reflects how the model will be used. Group related conversations, users, documents, or time windows when random row splitting would leak context. Version the source snapshot, filters, transformations, tokenizer assumptions, and split seed. Store immutable manifests and checksums so a reported result can be reproduced. Keep the test set access-controlled and change it deliberately, with an explanation of why new cases were added.
Inspect before training
Generate a human-readable sample of tokenized and rendered examples from every split. Check that roles, target boundaries, special tokens, truncation, labels, and structured fields look correct. Produce summary statistics for length, class distribution, language, source, and sensitive-data detection. Run a baseline evaluation with the untuned model and record representative failures. Only then start training. Dataset preparation is successful when another engineer can understand what the model will learn and why the test result should be trusted.