A decision tree predicts by asking a series of yes/no questions about the features, like a flowchart.
How a Tree Is Built
Starting with all the training data, the algorithm picks the feature and threshold that best separates the outcomes — for example, income < 40,000. It splits the data into two groups and repeats the process within each group, until a stopping rule is reached. Each final group, a leaf, predicts the most common class or the average value of its examples.
"Best" split is measured with criteria such as Gini impurity or entropy for classification and variance reduction for regression.
Strengths
- Easy to visualise and explain.
- Handle numeric and categorical features and non-linear relationships.
- Need little data preparation — no scaling required.
Weaknesses
- A deep tree overfits easily, memorising noise.
- Small changes in data can produce a very different tree (high variance).
- Predictions are step-like rather than smooth.
Controlling Overfitting
Limit max_depth, require a minimum number of samples per leaf, or prune the tree after growing it.
Where Trees Really Shine
Combined into ensembles — random forests and gradient boosting — trees become some of the most accurate models for tabular data. A single shallow tree remains valuable for explanation and quick insight.