10
$\begingroup$

A newbie question here. I am currently performing a nonparametric regression using the np package in R. I have 7 features and using a brute force approach I identified the best 3. But, soon I will have many more than 7 features!

My question is what are the current best methods for feature selection for nonparametric regression. And which if any packages implement the methods. Thank you.

$\endgroup$
  • 1
    $\begingroup$ what do you mean by "many more" 100 ? 1000 ? 10000? 100000 ? $\endgroup$ – robin girard Mar 7 '11 at 12:33
  • $\begingroup$ Probably I will have on the order of 100 features. But I have only a few minutes to make a decison on the best feature subset. $\endgroup$ – jmmcnew Mar 8 '11 at 18:34
  • 1
    $\begingroup$ Have you tried lasso or elastic net? packages: lasso, glmnet. Those methods can "select" some varibles on-the-go. $\endgroup$ – deps_stats Mar 12 '11 at 0:36
3
$\begingroup$

Unless identification of the most relevant variables is a key aim of the analysis, it is often better not to do any feature selection at all and use regularisation to prevent over-fitting. Feature selection is a tricky procedure and it is all too easy to over-fit the feature selection criterion as there are many degrees of freedom. LASSO and elastic net are a good compromise, the achieve sparsity via regularisation rather than via direct feature selection, so they are less prone to that particular form of over-fitting.

$\endgroup$
0
$\begingroup$

Lasso is indeed a good one. Simple things like starting with none, and adding them one by one sorted on 'usefullness' (via cross-validation) do also work quite well in practice. This is sometimes called stagewise feedforward selection.

Note that the subset selection problem is fairly independent on the type of classification / regression. It's just that nonparametric methods can be slow and therefore require more intelligent methods of selection.

The book 'The elements of statistical learning' from T. Hastie gives a nice overview.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.