A good model captures the real pattern in data and ignores the noise. Two failure modes pull in opposite directions.
Underfitting
The model is too simple to capture the pattern. It performs poorly on both training and validation data. A straight line fitted to clearly curved data underfits. Fixes include using a more flexible model, adding informative features or training longer.
Overfitting
The model is too flexible and learns the noise and coincidences in its training data. It scores very well on training data and noticeably worse on validation data. A decision tree allowed to grow without limit will often memorise every training example.
Reading the Gap
| Training score | Validation score | Diagnosis |
|---|---|---|
| Low | Low | Underfitting |
| High | Close to training | Good fit |
| Very high | Much lower | Overfitting |
Fixing Overfitting
- Get more, and more varied, training data.
- Simplify the model or limit its complexity (tree depth, number of features).
- Add regularisation, which penalises extreme parameter values.
- Use early stopping or dropout for neural networks.
- Check for data leakage — a feature that secretly contains the answer produces suspiciously good scores that vanish in production.
The Bias–Variance Trade-off
Underfitting is high bias (wrong assumptions); overfitting is high variance (too sensitive to the particular sample). Good modelling finds the balance.