Training with Backpropagation
Understand, at a conceptual and mechanical level, how a network learns from its mistakes.
Loss: measuring how wrong the network is
Training starts by defining a loss function that scores how far the network's prediction is from the true label — mean squared error for regression, cross-entropy for classification. This single number is what the entire training process tries to minimize. A well-defined loss function is the actual objective the network optimizes; everything else in training exists to reduce it efficiently.
Gradient descent in one paragraph
Gradient descent adjusts each weight in the direction that most reduces the loss, by a step size controlled by the learning rate. Imagine standing on a hilly loss surface in the fog and taking small steps downhill based only on the slope under your feet — that is gradient descent, repeated for every weight in the network, for every batch of training examples, over many passes (epochs) through the data.
import numpy as np
w = 0.5 # a single weight, simplified example
x = 2.0
y_true = 3.0
learning_rate = 0.1
for step in range(5):
y_pred = w * x
loss = (y_pred - y_true) ** 2
gradient = 2 * (y_pred - y_true) * x # d(loss)/d(w)
w -= learning_rate * gradient
print(f"step {step}: w={w:.3f} loss={loss:.3f}")Backpropagation: computing the gradient efficiently
Backpropagation is the algorithm that computes how much each individual weight contributed to the final loss, by applying the chain rule of calculus backward from the output layer to the input layer. It is what makes training deep networks computationally feasible — without it, computing each weight's gradient independently would be prohibitively expensive for networks with millions of parameters.
Learning rate and batch size
The learning rate controls how large each weight update is; too high and training can diverge or oscillate wildly, too low and training can take an impractically long time or get stuck in a poor local region. Batch size controls how many examples are averaged before each update: small batches give noisier but more frequent updates, while large batches give smoother but more computationally expensive updates. Both are tuned experimentally, not derived analytically.
Optimizers beyond plain gradient descent
Plain gradient descent is rarely used directly in practice. Momentum-based optimizers like Adam adaptively adjust the effective learning rate for each individual weight based on the history of its recent gradients, which usually trains faster and more reliably than a fixed learning rate. Choosing an optimizer is generally a low-effort decision — Adam or a close variant is a reasonable default for most problems — compared to getting the data and architecture right.
Practical exercise
Using the tiny two-layer network from How Neural Networks Work, manually perturb one weight by a small amount, recompute the forward pass, and observe how much the loss changes. Divide the change in loss by the change in the weight — that ratio is an approximation of the gradient backpropagation would compute exactly. This numerical-gradient exercise is also a real debugging technique engineers use to sanity-check a backpropagation implementation.
Sources and further reading
These primary or specialist references informed the concepts in this guide. Product details can change, so verify current documentation before implementation.