Feature selection for classification, controlling for sub-population

Question

I have a bunch of points that belong to one of population P1, P2, ... Pn AND to class A or B.

Within each population I'll be doing classification between A and B, and I want to select features that discriminates the best between A and B. Now, my features are also correlated with population membership, but I don't care about that, I only want to know how well the measures discriminate within a population.

So, is there a "proper" way to measure that? Within a population, I can just check how much the metric is correlated with class membership (biserial correlation), but if I get the correlation on the whole population, Simpson's paradox gets in the way and I may end up measuring the wrong thing.

I could just calculate correlations for each population, and average those or something, but that sounds ugly to me (this is for proramming real-world applications, not for publishing in a paper, so I don't mind if my method isn't that rigorous as long as the result is OK).

So, what's the proper way to control for population membership?

Answer 1

Regarding "Calculating one measure across all populations":

If you want to find one set of features so that the discriminative measure is good for each population, then calculating an aggregation of the measures (e.g. avg) is certainly a way to go. Depending on what you want to achieve, you can vary how this aggregation is calculated. Examples:

• All populations are weighted equal => avg
• The quality shall be as good as possible for every population, but is not allowed to drop beneath a certain threshold => min
• The population with greater size shall be weighted more => weighted avg with weight = size of population

Regarding "General Model-Building":

As the hands-on machine learner that I am, I recommend to

• merge the populations and calculate a good feature subset (this is what you want to do, if I understood you correctly)
• merge the populations + add a feature "population", which contains something like the "population id" and than calculate a good feature subset
• calculate a feature subset for each population separately

This will show you whether the division into subpopulations is really an issue or can be skipped anyways. Sometimes correlations between predictors and population do not influence the correlation between predictors and response.

Answer 2

A bit more research on Wikipedia indicates that ANCOVA may be what I need.

Answer 3

I think that linear discriminant analysis is a better option:

http://en.wikipedia.org/wiki/Linear_discriminant_analysis

-Ralph Winters

Answer 4

If you're only classifying within a population, don't worry about it. Just make sure you present your results in separate tables (or charts or whatever), one for each class.

If you're trying to classify observations who's class you do not know, then you need to be more careful.

Also, try a randomForest if you know how to use R, because I love random forests.