A lender’s database might hold 300 columns about each applicant, many of them redundant, noisy or expensive to collect. Feature selection narrows that to, say, 25 that carry most of the predictive signal. The final model then reads only those 25 fields, which makes it cheaper to run, easier to explain and less likely to latch onto coincidences in the training data.
Methods fall into three groups. Filter methods score each column on its own, for example by how strongly it relates to the outcome. Wrapper methods, such as recursive feature elimination, train a model, drop the weakest columns and repeat. Embedded methods select while training: an L1 penalty, a kind of regularization, pushes the weights of unhelpful columns to exactly zero.
The common mistake is choosing features using all the data, including the rows later used to test the model. The test score then looks better than real performance will be, because the test data already shaped the choice. Select features on training data only. Dropping a sensitive column, such as age, also doesn’t guarantee the model ignores it, because other columns can stand in for it. Feature extraction is the different job of creating new measurements from raw data.
Sources
- scikit-learn: Feature selection — Covers removing low-variance features, univariate statistical selection, recursive feature elimination and selection from a model's coefficients or importances, such as L1-penalized models; checked October 24, 2026.
- scikit-learn: Common pitfalls, data leakage during pre-processing — Uses feature selection on the full dataset before splitting as its example of data leakage that inflates test scores, and recommends fitting it on training data only inside a pipeline; checked October 24, 2026.