Gradient boosting builds an ensemble of small decision trees one after another, each new tree correcting the errors of those before it.
How It Works
- Start with a simple prediction, such as the average target value.
- Compute the errors (more precisely, the gradient of the loss) on the training data.
- Fit a small tree to predict those errors.
- Add the tree's predictions, scaled by a learning rate, to the model.
- Repeat for hundreds or thousands of rounds.
Because each tree focuses on what the ensemble still gets wrong, the model steadily improves.
Popular Implementations
- XGBoost, LightGBM and CatBoost — fast, well-engineered libraries widely used in industry and competitions.
- scikit-learn's
HistGradientBoostingClassifierand…Regressor— a fast built-in option.
Key Hyperparameters
- Learning rate — smaller values need more trees but generalise better.
- Number of trees — use early stopping on a validation set to choose.
- Tree depth / number of leaves — controls complexity.
- Subsampling of rows and columns — adds randomness that reduces overfitting.
Strengths and Weaknesses
Gradient boosting is usually the most accurate choice for structured, tabular data and handles missing values and mixed feature types well. It needs more careful tuning than a random forest and can overfit if trained too long.
Tip
Always use early stopping against a validation set rather than guessing the number of trees.