# Feature selection for classification, controlling for sub-population

I have a bunch of points that belong to one of population P1, P2, ... Pn AND to class A or B.

Within each population I'll be doing classification between A and B, and I want to select features that discriminates the best between A and B. Now, my features are also correlated with population membership, but I don't care about that, I only want to know how well the measures discriminate within a population.

So, is there a "proper" way to measure that? Within a population, I can just check how much the metric is correlated with class membership (biserial correlation), but if I get the correlation on the whole population, Simpson's paradox gets in the way and I may end up measuring the wrong thing.

I could just calculate correlations for each population, and average those or something, but that sounds ugly to me (this is for proramming real-world applications, not for publishing in a paper, so I don't mind if my method isn't that rigorous as long as the result is OK).

So, what's the proper way to control for population membership?

-
 It sounds as if Simpson's paradox will not be in play, since you are going to analyze within populations only....Do you care about collinearity, or about control of other predictors while checking a given predictor's relationship with A-or-B class? If so, you'll want to look into either discriminant analysis or (probably) the easier method of logistic regression. But it's not trivial either, and the conceptual issues involving control and variable selection often trip people up just as the technical, computational ones do. – rolando2 Mar 3 '11 at 17:07 If you're only classifying within a population, don't worry about it. Just make sure you present your results in separate tables (or charts or whatever), one for each class. If you're trying to classify observations who's class you do not know, then you need to be more careful. – Zach Mar 3 '11 at 19:31 @Zach, @rolando2 Make answers! – mbq♦ Mar 3 '11 at 20:42

Regarding "Calculating one measure across all populations":

If you want to find one set of features so that the discriminative measure is good for each population, then calculating an aggregation of the measures (e.g. avg) is certainly a way to go. Depending on what you want to achieve, you can vary how this aggregation is calculated. Examples:

• All populations are weighted equal => avg
• The quality shall be as good as possible for every population, but is not allowed to drop beneath a certain threshold => min
• The population with greater size shall be weighted more => weighted avg with weight = size of population

Regarding "General Model-Building":

As the hands-on machine learner that I am, I recommend to

• merge the populations and calculate a good feature subset (this is what you want to do, if I understood you correctly)
• merge the populations + add a feature "population", which contains something like the "population id" and than calculate a good feature subset
• calculate a feature subset for each population separately

This will show you whether the division into subpopulations is really an issue or can be skipped anyways. Sometimes correlations between predictors and population do not influence the correlation between predictors and response.

-

A bit more research on Wikipedia indicates that ANCOVA may be what I need.

-

I think that linear discriminant analysis is a better option:

http://en.wikipedia.org/wiki/Linear_discriminant_analysis

-Ralph Winters

-

If you're only classifying within a population, don't worry about it. Just make sure you present your results in separate tables (or charts or whatever), one for each class.

If you're trying to classify observations who's class you do not know, then you need to be more careful.

Also, try a randomForest if you know how to use R, because I love random forests.

-