A single train/validation split can be lucky or unlucky. Cross-validation averages over several splits for a steadier estimate.
k-Fold Cross-Validation
- Split the training data into k equal parts (folds), often 5 or 10.
- Train on k−1 folds and evaluate on the remaining fold.
- Repeat k times, holding out each fold once.
- Report the mean and spread of the k scores.
The spread matters: if scores vary widely between folds, the model's performance is unstable.
Variants
- Stratified k-fold keeps class proportions the same in every fold — important for imbalanced data.
- Group k-fold keeps all rows from one group (a patient, a customer, a store) in the same fold, preventing leakage between related records.
- Time-series split always trains on earlier data and validates on later data.
- Leave-one-out uses every single row as a validation set once; thorough but expensive.
Using It for Tuning
Cross-validation is the standard way to compare models and tune hyperparameters (for example with GridSearchCV). Keep a separate test set untouched until the very end.
Avoid Leakage
Any preprocessing learned from data — scaling, imputation, feature selection — must happen inside each fold. Wrapping steps in a scikit-learn Pipeline ensures this.
from sklearn.model_selection import cross_val_score
scores = cross_val_score(pipeline, X, y, cv=5, scoring="f1")
print(scores.mean(), scores.std())