Which one should be applied first: data sampling or dimensionality reduction?

I am working on binary classification problem. Data set is very large and highly imbalanced.
Data dimensionality is also very high. Now I want to balance data by under-sampling the majority class, and I also want to reduce data dimensionality by applying PCA, etc...

So my question is that which one should be applied first: data sampling or dimensionality reduction? Please also give argument in favor of your answer.

• Applying PCA to reduce dimension in a binary classification problem is nonsence. The directions of highest variablility are not necessarily the direction of highest separation, could be the opposite or anything. Do you need examples to be convinced? Feb 16, 2011 at 7:14
• @robin I'm not convinced it's nonsense; it depends on your aims. It's not going to identify the linear combinations of covariates most strongly associated with the binary outcome, but that may not be the aim. E.g. PCA is popular in analysing diet-disease studies as a step in reducing the huge amount of info collected from dietary diaries. One advantage is that as the principal components are chosen before their association with the binary outcome is examined, the analysis is free of charges of 'data snooping'. Feb 16, 2011 at 9:06
• @onestop I said it is nonsence in a binary classification problem. I assume that a binary classification problem is a problem where you need to predict a class that can be zero or one (binary) according to explanatory variables. If the aim is not to identify the relation (not necessarily linear) between covariates and binary outcome, I am wrong about my definition of binary classification. It is not because an algorithm is "popular" that it is a good idea to use it without thinking if it suites to your problem. Feb 16, 2011 at 9:19
• @robin Fair enough, I assume you wouldn't class the type of diet-disease study i'm thinking of as a binary classification problem then, and you're probably correct. Feb 16, 2011 at 10:04

Generally, you want your training and validation data sets be separate as much as possible. Ideally, the validation set data would have been obtained only after the model has been trained. If you perform dimensionality reduction before splitting your data to separate sets, you break this isolation between the training and the validation and you won't be sure whether the dimensionality reduction process was over-fitted until your model is tested in real life.

Having said that, there are cases, where efficient separation to training, testing and validation sets is not feasible and other sampling techniques, such as cross validation, leave k out etc are used. In these cases reducing the dimensionality before the sampling might be the right approach.

Do the dimensionality reduction first: Your error in estimating the principal components will be smaller due to the larger sample (your Corr/Cov-matrix used in PCA has to be estimated!).

The other way around only makes sense for computational reasons.

Devil's advocate: I could imagine the principal components differing depending on who's sampled. I'd think this validity issue would take precedence over the precision issue Richard points out.

You should perform sampling and dimensionality reduction in combination. The best way to do this is undersample the majority class, and run a decision tree. It is the best variable selector you can imagine. Perform this a number of times (each time another sample). The result will be a number of list of candidate predictors. And ... yes : combination of your decision trees is already a great model. Find out why decision trees is the best data mining algorithm at http://bit.ly/a2qDWJ

• I visited the link and read your praise of the decision tree. Have you considered the way this method treats all main effects (save the first) as if they were interaction effects? I have found that highly misleading and counterproductive on many occasions. Feb 25, 2011 at 19:06