This is perhaps a typical setup in Bioinformatics: we need to build a model to predict a dependent biological variable (say $y$), given a large set of (usually genomic) features $X$ (which might not all be relevant). The size of the training set, $n$, is usually much smaller than the number of (potential) features, $|X|$. The trick is then to find the right set of features which most likely be the most predictive of the response variable $y$. Now, suppose using some prior biological knowledge and pre-processing, I have selected a subset of $X$, say $X'$ that is deemed to be the more relevant features to predict $y$ (note that: in this case, we still have $|X'|>n$). I want to know how predictive it is against $y$ in the training set. Of course, I can select a certain statistical/machine learning method, say stepwise regression or LASSO (or any other regression method that can handle $|X'|> n$), and then do cross-validation within the training set to get an idea of how predictive $X'$ is towards $y$. However, I was wondering whether I can do this in a non-model-specific way, i.e., a general metric to say how predictive $X'$ is for any model. I don't think simple correlation really captures what I need, since there can be many high-correlating variables which might not really be good predictors (spurious correlation).
-
$\begingroup$ There is an excellent and recent CV discussion of this specific issue including commentary by Frank Harrell, chair of Biostatistics at Vanderbilt here ... stats.stackexchange.com/questions/179693/… $\endgroup$– user78229Commented Nov 10, 2015 at 11:07
2 Answers
I can direct you to the Fast Feature Selection based on R^2-values, based on Pearson Correlation Coefficient. This method was introduced for multidimensional MEG or EEG data, however it suits any binary machine learning problem. Put simply: it computes correlations of data and labels, then sorts obtained values, finally only the best scoring features are selected.
I implemented the approach for MATLAB and LIBSVM found at github. In my implementation you can define the amount of features to be selected, e.g. 100, 1000 or everything above the mean of scoring values.
-
$\begingroup$ Hi @dthettich, thanks for the suggestion. Can you comment on the applicability of this method for regression problems? $\endgroup$ Commented Nov 10, 2015 at 11:19
-
$\begingroup$ You're welcome. Unfortunately, the method in its present form requires two discrete labels. Yet maybe there is a possibility for you to extract meaningful features by "Mutual Information". $\endgroup$ Commented Nov 10, 2015 at 12:13
Accurately determining the predictive power of a set of features with respect to a dependent variable is a tricky problem, and there is no universal metric which can tell you exactly that.
You already selected a group of features $X'$ that you want to use, but it is in general common to use a feature selection method to determine such a set. Since you already have $X'$ you can instead use some of the metrics of these feature selection methods to determine the usefulness of your features in terms of prediction.
I personally prefer Correlation-based Feature Subset Selection which evaluates a set of features based on its correlation with respect to the depend variable, and the inter-correlation (i.e. redundancy) between individual features in the set:
http://www.cs.waikato.ac.nz/~mhall/thesis.pdf
Other metrics that may be useful are the information gain, or gain-ratio (commonly used in decision trees) to determine the entropy of the data after splitting the data with respect to a feature
