I have a large universe of features, and potentially a large universe of targets that I want to use them for. I need to construct some kind of summary stats that ranks the features by their relevance for each target, or perhaps state if a particular feature is irrelevant for a particular target, or maybe very good for a given target.

How would I go about doing this? Any ideas/references?

I am specifically interested in doing regression(rather than classification).

  • $\begingroup$ Try Logistic Regression with L2 or L1 regularization. The latter will prune the irrelevant features while the former will drive them close to 0. $\endgroup$ – Vladislavs Dovgalecs Oct 29 '15 at 18:04
  • $\begingroup$ I am doing regression. $\endgroup$ – The Baron Oct 30 '15 at 18:20

You may want to have a look at this very readable article (especially the introduction): http://www.jmlr.org/papers/volume3/guyon03a/guyon03a.pdf

The article first outlines basic questions that give an easy-to-grasp horizon when and how to proceed with feature selection. Furthermore, it gives an overview about single and multiple variable selection methods, most of which are available off-the-shelf in the usual ML libraries.

As you note that you have a "large universe of features", section 5 on dimensionality reduction gives a good start to get "more informative" features to be fed into classifiers that do not just find linear connections.

  • $\begingroup$ Please explain a bit more about the concept in the answer. Link-only answers are generally discouraged. $\endgroup$ – Dawny33 Oct 30 '15 at 6:09
  • $\begingroup$ @Dawny33 I would have commented on the question if I was allowed to, so I hope my motivation to refer to that paper is clearer now... $\endgroup$ – jmaxx Oct 30 '15 at 6:22
  • $\begingroup$ Yeah, now the answer looks good :) $\endgroup$ – Dawny33 Oct 30 '15 at 6:25
  • $\begingroup$ I am interested in regression specifically. So, is ranking by correlation the safe, common-sense approach, based on this paper? $\endgroup$ – The Baron Oct 30 '15 at 18:11
  • $\begingroup$ @TheBaron That is a starting point, however it may be insufficient. If you have to variables A and B as input and an output C, their relation may be C = A XOR B, which cannot be resolved using linear measures such as correlation. $\endgroup$ – jmaxx Oct 31 '15 at 5:28

Following are some of the methods which can be used :

1. Subset selection : Identifies subset of predictors that are related to the response. This can be accomplished using best subset selection or stepwise subset selection methods.

2. Shrinkage methods : Coefficients of predictors weakly related to response are shrunken towards zero. Ridge and Lasso regression can be used for this.

3. Dimension reduction : Find the predictors which are linearly correlated to other predictors. Can be done using PCA (Principal component analysis).

For more details and worked out examples in R you can refer to chapter 6 from "Introduction to Statistical Learning", which can be downloaded for free from here -> http://www-bcf.usc.edu/~gareth/ISL/

  • $\begingroup$ Thanks for your input. How would I use PCA for feature selection? $\endgroup$ – The Baron Oct 30 '15 at 14:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.