I have a large universe of features, and potentially a large universe of targets that I want to use them for. I need to construct some kind of summary stats that ranks the features by their relevance for each target, or perhaps state if a particular feature is irrelevant for a particular target, or maybe very good for a given target.

How would I go about doing this? Any ideas/references?

I am specifically interested in doing regression(rather than classification).

• Try Logistic Regression with L2 or L1 regularization. The latter will prune the irrelevant features while the former will drive them close to 0. – Vladislavs Dovgalecs Oct 29 '15 at 18:04
• I am doing regression. – The Baron Oct 30 '15 at 18:20

You may want to have a look at this very readable article (especially the introduction): http://www.jmlr.org/papers/volume3/guyon03a/guyon03a.pdf

The article first outlines basic questions that give an easy-to-grasp horizon when and how to proceed with feature selection. Furthermore, it gives an overview about single and multiple variable selection methods, most of which are available off-the-shelf in the usual ML libraries.

As you note that you have a "large universe of features", section 5 on dimensionality reduction gives a good start to get "more informative" features to be fed into classifiers that do not just find linear connections.

• @Dawny33 I would have commented on the question if I was allowed to, so I hope my motivation to refer to that paper is clearer now... – jmaxx Oct 30 '15 at 6:22
• Yeah, now the answer looks good :) – Dawny33 Oct 30 '15 at 6:25
• I am interested in regression specifically. So, is ranking by correlation the safe, common-sense approach, based on this paper? – The Baron Oct 30 '15 at 18:11
• @TheBaron That is a starting point, however it may be insufficient. If you have to variables A and B as input and an output C, their relation may be C = A XOR B, which cannot be resolved using linear measures such as correlation. – jmaxx Oct 31 '15 at 5:28

Following are some of the methods which can be used :

1. Subset selection : Identifies subset of predictors that are related to the response. This can be accomplished using best subset selection or stepwise subset selection methods.

2. Shrinkage methods : Coefficients of predictors weakly related to response are shrunken towards zero. Ridge and Lasso regression can be used for this.

3. Dimension reduction : Find the predictors which are linearly correlated to other predictors. Can be done using PCA (Principal component analysis).

For more details and worked out examples in R you can refer to chapter 6 from "Introduction to Statistical Learning", which can be downloaded for free from here -> http://www-bcf.usc.edu/~gareth/ISL/

• Thanks for your input. How would I use PCA for feature selection? – The Baron Oct 30 '15 at 14:38