Model Evaluation Metrics Explained
Understand accuracy, precision, recall, F1, and RMSE well enough to know when each one is lying to you.
Why accuracy alone is dangerous
Accuracy — the percentage of predictions that are correct — is intuitive but misleading on imbalanced data. If 99% of transactions are legitimate, a model that always predicts "not fraud" scores 99% accuracy while being completely useless. This is the single most common evaluation mistake in applied machine learning, and it is why fraud, medical, and security models are almost never evaluated on accuracy alone.
Precision and recall
Precision answers "of everything the model flagged as positive, how much was actually positive?" Recall answers "of everything that was actually positive, how much did the model catch?" A spam filter tuned for high precision rarely marks real mail as spam but might miss some spam. A spam filter tuned for high recall catches nearly all spam but risks flagging real mail too. The right balance depends entirely on which mistake — a false positive or a false negative — costs more in your specific product.
from sklearn.metrics import precision_score, recall_score, f1_score, confusion_matrix
y_true = [1, 1, 0, 1, 0, 0, 1, 0, 1, 0]
y_pred = [1, 0, 0, 1, 0, 1, 1, 0, 0, 0]
print("Confusion matrix:\n", confusion_matrix(y_true, y_pred))
print("Precision:", precision_score(y_true, y_pred))
print("Recall:", recall_score(y_true, y_pred))
print("F1:", f1_score(y_true, y_pred))The confusion matrix
A confusion matrix lays out true positives, false positives, true negatives, and false negatives in a 2x2 grid for binary classification. Every other classification metric is derived from these four numbers. Reading the raw matrix before looking at any summary metric is the fastest way to understand what kind of mistakes a model is actually making, rather than trusting a single number that averages two very different failure modes together.
F1 score and when to use it
The F1 score is the harmonic mean of precision and recall, giving a single number that penalizes models which sacrifice one entirely for the other. It is a reasonable default when false positives and false negatives are roughly equally costly and the classes are imbalanced. It is a poor choice when the costs are clearly asymmetric — in that case, report precision and recall separately, or use a cost-weighted metric that reflects the real business impact of each error type.
Regression metrics: MAE, MSE, and RMSE
Mean absolute error (MAE) reports the average size of the error in the same units as the target, which makes it easy to explain. Mean squared error (MSE) squares each error before averaging, which punishes large misses much more than small ones. Root mean squared error (RMSE) takes the square root of MSE to bring the units back to the original scale while keeping that sensitivity to large errors. Choosing between them depends on whether large, rare errors are especially costly in your use case.
Practical exercise
Take a hypothetical spam filter with 950 real emails and 50 spam emails. Suppose it flags 40 emails as spam, of which 30 are correctly spam and 10 are real emails wrongly flagged, and it misses 20 actual spam emails. Calculate precision, recall, and F1 by hand from these numbers before checking with a library, then decide whether you would ship this model for a personal inbox versus a corporate compliance system.
Sources and further reading
These primary or specialist references informed the concepts in this guide. Product details can change, so verify current documentation before implementation.