PCA and random forests For a recent Kaggle competition, I (manually) defined 10 additional features for my training set, which would then be used to train a random forests classifier. I decided to run PCA on the dataset with the new features, to see how they compared to each other. I found that ~98% of the variance was carried by the first component (the first eigenvector). I then trained the classifier multiple times, adding one feature at a time, and used cross-validation and RMS error to compare the quality of the classification. I found that the classifications improved with each additional feature, and that the final result (with all 10 new features) was far better than the first run with (say) 2 features. 


*

*Given that PCA claimed ~98% of the variance was in the first component of my dataset, why did the quality of the classifications improve so much? 

*Would this hold true for other classifiers? RF scales across multiple cores, so it's much faster to train than (say) SVM.

*What if I had transformed the dataset into the "PCA" space, and run the classifier on the transformed space. How would my results change?
 A: When doing predictive modeling, you are trying to explain the variation in the response, not the variation in the features.  There is no reason to believe that cramming as much of the feature variation into a single new feature will capture a large amount of the predictive power of the features as a whole.
This is often explained as the difference between Principal Component Regression instead of Partial Least Squares.
A: The first principal component is a linear combination of all your features.  The fact that it explains almost all the variability just means that most of the coefficients of the variables in the first principal component are significant.  
Now the classification trees you generate are a bit of a different animal too. They do binary splits on continuous variables that best separate the categories you want to classify.  That is not exactly the same as finding orthogonal linear combinations of continuous variables that give the direction of greatest variance.  In fact we have recently discussed a paper on CV where PCA was used for cluster analysis and the author(s) found that there are situations wher best separation is found not in the 1st few principal components but rather in the last ones.
