More features are not always better. Irrelevant or redundant features add noise, slow training and increase the risk of overfitting.
Why Select Features
- Better generalisation with less noise.
- Faster training and prediction.
- Simpler, more explainable models.
- Lower cost of collecting and maintaining data.
Filter Methods
Score each feature independently of any model, then keep the best:
- correlation with the target (for numeric targets);
- mutual information;
- chi-squared tests for categorical data;
- dropping near-constant or highly correlated duplicate features.
Fast, but they ignore how features interact.
Wrapper Methods
Train models with different feature subsets and keep the best-performing set. Recursive feature elimination (RFE) repeatedly drops the weakest feature. Accurate but computationally expensive.
Embedded Methods
Selection happens during training:
- Lasso drives weak coefficients to zero.
- Tree ensembles provide importance scores.
Avoid Leakage
Feature selection is a learned step. Do it inside cross-validation folds (for example within a pipeline), or the chosen features will reflect the validation data and inflate scores.
Use Domain Knowledge
Automated scores can't tell you whether a feature will be available at prediction time, is legally usable, or is a proxy for a sensitive attribute. Review selected features with people who know the data.