Activation functions introduce non-linearity into neural networks. Without them, a deep network would behave like a single linear model.
Common Functions
- ReLU: outputs the input if positive, otherwise zero. Simple, fast and widely used in hidden layers.
- Leaky ReLU: like ReLU but with a small slope for negative inputs, avoiding "dead" units that never activate.
- GELU and SiLU (Swish): smooth alternatives common in transformers and modern architectures.
- Sigmoid: squashes values to 0–1. Used for binary outputs and gates; prone to vanishing gradients in deep hidden layers.
- Tanh: squashes values to −1 to 1; zero-centred.
- Softmax: turns a vector of scores into probabilities summing to 1, for multi-class outputs.
Choosing
- Hidden layers: ReLU or GELU-family functions are safe defaults.
- Binary or multi-label outputs: sigmoid.
- Multi-class outputs: softmax.
- Regression outputs: usually no activation (linear).
Vanishing Gradients
Functions that saturate, like sigmoid, produce tiny gradients, slowing learning in deep networks — a key reason ReLU-family functions became standard.