Overfitting, Underfitting, and Regularization
Diagnose why a model performs well in training and poorly in the real world, and learn the standard techniques to fix it.
Overfitting in plain terms
Overfitting happens when a model learns the noise and idiosyncrasies of its training data instead of the general pattern behind it. It performs extremely well on data it has already seen and noticeably worse on new data. A telltale sign is a large, growing gap between training accuracy and validation accuracy as training continues — the model is memorizing rather than generalizing.
Underfitting in plain terms
Underfitting is the opposite failure: the model is too simple to capture the real pattern, so it performs poorly on both training and validation data. A straight line trying to fit a clearly curved relationship is a classic underfitting example. Underfitting is usually easier to diagnose than overfitting because both scores are visibly bad, not just the validation score.
from sklearn.linear_model import Ridge
from sklearn.preprocessing import PolynomialFeatures
from sklearn.pipeline import make_pipeline
for degree, alpha in [(1, 0), (3, 1.0), (12, 0.0)]:
model = make_pipeline(PolynomialFeatures(degree), Ridge(alpha=alpha))
model.fit(X_train, y_train)
print(f"degree={degree} alpha={alpha} -> "
f"train R^2={model.score(X_train, y_train):.2f}, "
f"val R^2={model.score(X_val, y_val):.2f}")The bias-variance trade-off
Bias is the error from a model being too simple to represent the true relationship; variance is the error from a model being too sensitive to the specific training data it happened to see. Simple models tend to have high bias and low variance (they underfit); complex models tend to have low bias and high variance (they overfit). Most of model tuning is a search for the point on this trade-off that generalizes best.
Regularization techniques
Regularization adds a penalty for model complexity so the training process favors simpler, more general solutions. L1 (Lasso) regularization can shrink unhelpful feature weights to exactly zero, effectively performing feature selection. L2 (Ridge) regularization shrinks all weights smoothly without zeroing them out. Dropout, used in neural networks, randomly disables a fraction of neurons during training so the network cannot over-rely on any single path.
Other practical fixes
Beyond formal regularization, the most effective fixes for overfitting are often simpler: gather more training data, remove noisy or irrelevant features, use early stopping to halt training once validation performance stops improving, and use cross-validation instead of a single train/test split so the evaluation itself is more stable. For underfitting, the fix usually runs the other direction: add features, use a more expressive model, or train for longer.
Practical exercise
Plot training accuracy and validation accuracy against training time (or model complexity) for any model you have access to, even a toy scikit-learn example. Identify the point where the curves start diverging — that is roughly where overfitting begins. Then apply one regularization technique (like increasing L2 strength) and re-plot to see whether the gap narrows and where validation performance peaks.
Sources and further reading
These primary or specialist references informed the concepts in this guide. Product details can change, so verify current documentation before implementation.