Random Forests

A random forest combines many decision trees and averages their predictions. The result is far more robust than any single tree.

How It Works

1. Bootstrap sampling: each tree is trained on a random sample of the training rows, drawn with replacement.
2. Random feature selection: at each split, a tree considers only a random subset of features.
3. Aggregation: for classification, trees vote; for regression, their predictions are averaged.

The randomness makes the trees different from one another, so their individual errors tend to cancel out. This reduces overfitting compared with a single deep tree.

Strengths

• Strong accuracy on tabular data with minimal tuning.
• Robust to outliers and irrelevant features.
• No feature scaling needed.
• Provides feature importance scores and an out-of-bag error estimate.

Weaknesses

• Less interpretable than a single tree.
• Large forests can be slow and memory-hungry at prediction time.
• Often slightly less accurate than well-tuned gradient boosting.

Key Hyperparameters

• n_estimators — number of trees; more is better up to a point.
• max_features — features considered per split.
• max_depth and min_samples_leaf — tree complexity.

A Caution on Feature Importance

Default impurity-based importance can favour features with many unique values. Permutation importance on held-out data is more reliable.

from sklearn.ensemble import RandomForestClassifier
rf = RandomForestClassifier(n_estimators=300, random_state=0).fit(X_train, y_train)