Activation Functions Explained
Compare sigmoid, ReLU, and softmax, and understand why the choice of activation function shapes what a network can learn.
What an activation function does
An activation function takes the weighted sum coming into a neuron and reshapes it, usually introducing nonlinearity and sometimes bounding the output to a fixed range. Without one, a "deep" network would behave like a single linear layer no matter how many layers it had. The choice of activation function affects training speed, stability, and what kind of output the layer can represent.
Sigmoid and its problems
The sigmoid function squashes any input into a range between 0 and 1, which made it a natural early choice for representing probabilities. Its major weakness is the vanishing gradient problem: for very large or very negative inputs, the function is nearly flat, so the gradient used to update weights during training becomes tiny, and deep networks using sigmoid throughout can become extremely slow or fail to train past a handful of layers.
import numpy as np
def sigmoid(x): return 1 / (1 + np.exp(-x))
def relu(x): return np.maximum(0, x)
def sigmoid_grad(x): s = sigmoid(x); return s * (1 - s)
x = np.array([-6, -2, 0, 2, 6], dtype=float)
print("sigmoid:", sigmoid(x))
print("sigmoid gradient:", sigmoid_grad(x)) # notice how small it gets at the extremes
print("relu:", relu(x))ReLU and its variants
The Rectified Linear Unit (ReLU) simply outputs the input if it is positive, and zero otherwise. It is cheap to compute and avoids the vanishing-gradient problem for positive inputs, which is why it became the default choice for hidden layers in most modern networks. Its own weakness, the dying ReLU problem, happens when a neuron gets stuck outputting zero for every input; variants like Leaky ReLU allow a small negative slope to keep those neurons trainable.
Softmax for multiclass output
Softmax is almost always used on the final layer of a multiclass classifier. It converts a vector of raw scores into a probability distribution — all values between 0 and 1, summing to exactly 1 — so the network's output can be interpreted as "this input is 82% likely to be class A, 12% class B, 6% class C." It is a poor choice for hidden layers because it forces outputs to compete with each other, which is not useful mid-network.
Choosing an activation function in practice
A common practical default is ReLU (or a variant) for all hidden layers, sigmoid for a single-output binary classification head, and softmax for a multiclass classification head. Modern large language models often use smoother variants like GELU or SiLU in their hidden layers, which trade a small amount of compute cost for better gradient behavior at the scale these models operate at.
Practical exercise
Plot sigmoid, ReLU, and tanh on the same graph for inputs from -10 to 10 (any spreadsheet or plotting tool works). Identify the input range where each function's slope is nearly zero — that is where a network using that activation is most prone to a vanishing gradient. This visual comparison makes the abstract vanishing gradient concept concrete before you encounter it in a training-instability debugging session.
Sources and further reading
These primary or specialist references informed the concepts in this guide. Product details can change, so verify current documentation before implementation.