A model's prediction error can be broken into parts that help explain overfitting and underfitting.
Bias
Error from overly simple assumptions. A high-bias model misses real patterns — a straight line fitted to curved data. This is underfitting.
Variance
Error from sensitivity to the particular training data. A high-variance model fits noise and changes a lot with different samples. This is overfitting.
The Trade-Off
As complexity increases, bias usually falls and variance rises. The best model balances them to minimise total error on new data.
Diagnosing
- High training error and high validation error → high bias. Try more features or a more flexible model.
- Low training error, high validation error → high variance. Try more data, regularisation or a simpler model.
Tools for Balance
- Regularisation.
- Cross-validation to choose complexity.
- Ensembles: bagging reduces variance; boosting reduces bias.
- More training data reduces variance.
A Modern Twist
Very large neural networks can generalise well despite fitting training data perfectly — a phenomenon called double descent — so the classic picture is a guide, not a law.