Avoiding Common Neural Network Training Pitfalls
Recognize the most frequent, avoidable mistakes that make a neural network fail to train or fail to generalize.
Vanishing and exploding gradients
In very deep networks, gradients computed during backpropagation can shrink toward zero (vanishing) or grow uncontrollably large (exploding) as they propagate backward through many layers. Symptoms include a loss that barely moves for many epochs, or a loss that suddenly becomes NaN. Modern fixes include careful weight initialization, batch normalization, gradient clipping, and choosing activation functions like ReLU that are less prone to vanishing gradients than sigmoid or tanh.
Poor weight initialization
Initializing all weights to the same value (including zero) causes every neuron in a layer to learn an identical, redundant function, since they all receive the same gradient during training — a failure mode called symmetry that prevents the network from learning anything useful. Standard initialization schemes like Xavier or He initialization set the initial random weight scale based on the layer's size specifically to avoid both this symmetry problem and early vanishing or exploding activations.
Not normalizing inputs
Feeding a network raw, unscaled features — some ranging from 0 to 1, others from 0 to 1,000,000 — typically slows training significantly and can prevent convergence altogether, because the loss surface becomes extremely elongated in some directions. Standardizing or normalizing every input feature before training is one of the highest-leverage, lowest-effort fixes available and should be a default step, not an optional one.
Overfitting on small datasets
Deep networks have enormous capacity to memorize, and on a small dataset they will often do exactly that rather than learn a generalizable pattern, producing excellent training accuracy and poor validation accuracy. Dropout, data augmentation, early stopping, and simply using a smaller network are all standard countermeasures; the deeper fix is usually recognizing that a small dataset may call for a much simpler model, or a pretrained model fine-tuned rather than trained from scratch.
Ignoring validation curves
The single most common avoidable mistake is training a model and checking only the final training loss, without ever plotting training loss against validation loss over time. That curve reveals overfitting, underfitting, and instability far earlier and more clearly than any single end-of-training number, and should be the first thing checked whenever a model is not performing as expected.
Practical exercise
If you have trained any model before (even a scikit-learn one), pull up its training log or history and plot training loss and validation loss on the same chart across epochs or iterations. If you have not trained a model before, sketch by hand what a healthy training curve should look like versus an overfitting one and an underfitting one, labeling the point where you would stop training in each case.
Sources and further reading
These primary or specialist references informed the concepts in this guide. Product details can change, so verify current documentation before implementation.