Choosing the best featureset for prediction

Question

I have this N sets of features F each with $F_i$ number of features. All the feature sets have 20000 examples and we have 20,000 labels for them.

Lets say feature set $F_1$ has 10 features and feature set $F_2$ has 20 features. Each of these feature sets have 20000 examples. Now all these examples have labels.

So I want to decide which of these feature sets is useful for the classification. I don't want to create a classifier and decide because my labels are highly unbalanced.

I was thinking of doing this with canonical correlation. I will take one feature set $F_i$ and the label vector and do cannonical correlation, get the score.

Do that for all the feature sets and then sort them based upon the scores obtained. How would that be?

Answer 1

Unbalanced labels is no problem. Create a classifier, and for your evaluation measure, use a measure that works correctly for unbalanced data. In other words, don't use accuracy, use Area Under the ROC Curve.

Canonical correlation? Maybe. But that is essentially treating the problem as a kind of linear regression problem, and therefore you might reject features that are strong predictors of your outcomes but with a strongly nonlinear relationship.

Answer 2

That might work, but try to divide and conquer so that you reduce your large problem to many small problems.

First, get all of the $p$ features known, and all of the $n$ examples that are known, and construct an $n \times p$ data matrix $\mathbf{X}$. Then, answer the question: Which features predict class membership best using all the training data? This is then followed by breaking up the examples into folds to run CV, for which some groups claim you should identify the best features which best predict class membership of example in the fold left out of training. Other groups claim you can do what you are doing during CV feature identification.