0
$\begingroup$

I have a dataset with 25 features(columns). I'm trying to apply forward feature selection to my dataset, which will return a subset containing the best features. But the size of this subset, i.e. the number of selected features (say k), has to be specified a priori. How do I determine what is the best value of k for the given dataset.

$\endgroup$
1
$\begingroup$

Though this way of thinking is commonly believed, the data are incapable of delivering the information that allows selection of the 'right' variables. You didn't provide any motivation for why feature selection was desired. The sample size and distribution of $Y$ are also critical.

If you decide to use feature selection over just specifying a model (with restrictions imposed by unsupervised learning, i.e., data reduction, to deal with the dimensionality), be sure to bootstrap the entire process from scratch to get frightened about the instability of features selected.

There are rules of thumb about the selection of $k$ but these rules applied to pre-specified models. For example, if you pre-specified $k$ features where $k < \frac{m}{15}$ you may avoid overfitting. Here $m$ is the effective sample size. If $Y$ is binary, $m$ is the minimum of the number of events and the number of non-events. This is discussed at length in Regression Modeling Strategies.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.