Judging a model on the data it trained on is like grading a student on an exam they have already seen. To get an honest answer, data is split.
The Three Splits
- Training set — used to fit the model's parameters. Usually the largest share, often 60–80%.
- Validation set — used to compare models and tune settings such as tree depth or learning rate.
- Test set — used once, at the end, to estimate how the final model will perform on new data.
Why Keep the Test Set Separate?
Every time you look at test results and change something, you tune the model a little towards that particular test set. Do it often enough and the test score stops being an honest estimate. Make decisions on the validation set; report on the test set.
Cross-Validation
When data is limited, k-fold cross-validation splits the training data into k parts, trains k times holding a different part out each time, and averages the scores. It gives a more stable estimate than a single validation split.
Splitting Pitfalls
- Time-ordered data: train on the past, test on the future. A random split leaks future information.
- Grouped data: keep all records of the same person, patient or device on one side of the split.
- Imbalanced classes: use a stratified split so rare classes appear in every part.
- Duplicates: near-identical rows on both sides of the split inflate scores.