I have a dataset with 25 features(columns). I'm trying to apply forward feature selection to my dataset, which will return a subset containing the best features. But the size of this subset, i.e. the number of selected features (say k), has to be specified a priori. How do I determine what is the best value of k for the given dataset.
$\begingroup$
$\endgroup$
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
Though this way of thinking is commonly believed, the data are incapable of delivering the information that allows selection of the 'right' variables. You didn't provide any motivation for why feature selection was desired. The sample size and distribution of $Y$ are also critical.
If you decide to use feature selection over just specifying a model (with restrictions imposed by unsupervised learning, i.e., data reduction, to deal with the dimensionality), be sure to bootstrap the entire process from scratch to get frightened about the instability of features selected.
There are rules of thumb about the selection of $k$ but these rules applied to pre-specified models. For example, if you pre-specified $k$ features where $k < \frac{m}{15}$ you may avoid overfitting. Here $m$ is the effective sample size. If $Y$ is binary, $m$ is the minimum of the number of events and the number of non-events. This is discussed at length in Regression Modeling Strategies.
answered Jun 19, 2018 at 12:01