Which one should be applied first: data sampling or dimensionality reduction? I am working on binary classification problem. Data set is very large and highly imbalanced.
Data dimensionality is also very high.
Now I want to balance data by under-sampling the majority class, and I also want to reduce data dimensionality by applying PCA, etc... 
So my question is that which one should be applied first: data sampling or dimensionality reduction? 
Please also give argument in favor of your answer.
Thanks in advance 
 A: Do the dimensionality reduction first: Your error in estimating the principal components will be smaller due to the larger sample (your Corr/Cov-matrix used in PCA has to be estimated!).
The other way around only makes sense for computational reasons.
A: Generally, you want your training and validation data sets be separate as much as possible. Ideally, the validation set data would have been obtained only after the model has been trained. If you perform dimensionality reduction before splitting your data to separate sets, you break this isolation between the training and the validation and you won't be sure whether the dimensionality reduction process was over-fitted until your model is tested in real life.
Having said that, there are cases, where efficient separation to training, testing and validation sets is not feasible and other sampling techniques, such as cross validation, leave k out etc are used. In these cases reducing the dimensionality before the sampling might be the right approach.
A: Devil's advocate:  I could imagine the principal components differing depending on who's sampled.  I'd think this validity issue would take precedence over the precision issue Richard points out.
A: You should perform sampling and dimensionality reduction in combination.
The best way to do this is undersample the majority class, and run a decision tree.  It is the best variable selector you can imagine.
Perform this a number of times (each time another sample).  The result will be a number of list of candidate predictors.
And ... yes : combination of your decision trees is already a great model.
Find out why decision trees is the best data mining algorithm at http://bit.ly/a2qDWJ
