Advice on feature selection I have a large universe of features, and potentially a large universe of targets that I want to use them for. I need to construct some kind of summary stats that ranks the features by their relevance for each target, or perhaps state if a particular feature is irrelevant for a particular target, or maybe very good for a given target.
How would I go about doing this? Any ideas/references?
I am specifically interested in doing regression(rather than classification).
 A: You may want to have a look at this very readable article (especially the introduction): http://www.jmlr.org/papers/volume3/guyon03a/guyon03a.pdf
The article first outlines basic questions that give an easy-to-grasp horizon when and how to proceed with feature selection. Furthermore, it gives an overview about single and multiple variable selection methods, most of which are available off-the-shelf in the usual ML libraries. 
As you note that you have a "large universe of features", section 5 on dimensionality reduction gives a good start to get "more informative" features to be fed into classifiers that do not just find linear connections.
A: Following are some of the methods which can be used :
1. Subset selection : Identifies subset of predictors that are related to the response. This can be accomplished using best subset selection or stepwise subset selection methods.
2. Shrinkage methods : Coefficients of predictors weakly related to response are shrunken towards zero. Ridge and Lasso regression can be used for this.
3. Dimension reduction : Find the predictors which are linearly correlated to other predictors. Can be done using PCA (Principal component analysis).
For more details and worked out examples in R you can refer to chapter 6 from "Introduction to Statistical Learning", which can be downloaded for free from here -> http://www-bcf.usc.edu/~gareth/ISL/
