PCA and random forests

For a recent Kaggle competition, I (manually) defined 10 additional features for my training set, which would then be used to train a random forests classifier. I decided to run PCA on the dataset with the new features, to see how they compared to each other. I found that ~98% of the variance was carried by the first component (the first eigenvector). I then trained the classifier multiple times, adding one feature at a time, and used cross-validation and RMS error to compare the quality of the classification. I found that the classifications improved with each additional feature, and that the final result (with all 10 new features) was far better than the first run with (say) 2 features.

• Given that PCA claimed ~98% of the variance was in the first component of my dataset, why did the quality of the classifications improve so much?

• Would this hold true for other classifiers? RF scales across multiple cores, so it's much faster to train than (say) SVM.

• What if I had transformed the dataset into the "PCA" space, and run the classifier on the transformed space. How would my results change?

-
Did you normalize your data before running PCA? If I had to guess I'd think one of your features was on a much larger scale than the others... –  Marc Shivers Jul 23 '12 at 14:08
The PCA function automatically normalizes everything when doing the calculation. –  Vishal Jul 23 '12 at 14:16
Maybe that's just me, but could you clarify the following: your first step consisted in adding 10 (raw) features, one at a time, or did you work directly with PCA components? As stated, I understand this is the first case and you wonder whether you could work directly with results from PCA. In either case, did you apply the PCA on all variables, including new features, or just on the later? –  chl Jul 23 '12 at 16:18
I applied the PCA to the original matrix with the 10 additional features. I then trained the classifier, by adding in one feature at a time, so I could measure the incremental improvement provided with the addition of each feature. My question was what if I transformed the dataset (with the 10 new features) into the PCA space, and then ran the classifier directly on the dataset in the PCA space –  Vishal Jul 23 '12 at 17:16

When doing predictive modeling, you are trying to explain the variation in the response, not the variation in the features. There is no reason to believe that cramming as much of the feature variation into a single new feature will capture a large amount of the predictive power of the features as a whole.

This is often explained as the difference between Principal Component Regression instead of Partial Least Squares.

-
"There is no reason to believe that cramming as much of the feature variation into a single new feature will capture a large amount of the predictive power of the features as a whole." That was never the point, and doing this would result in a very confused classifier! The goal was to have a range of features, all illustrating different aspects of the dataset, with the intention of reducing the generalization error. The point of taking the PCA was see how different the features were. And my point of posting was that my features were not that different, yet the results of RF still improved. –  Vishal Jul 24 '12 at 14:54
The same logic can still apply. A new feature is highly colinnear with a prior feature can still bring more predictive power. Specifically for a randomForest: if the near duplicate feature is important in general, one version or another is now more likely to be selected as splitting candidates. –  Shea Parkes Jul 25 '12 at 3:14
This begs the follow-up question, how do you a-priori select features for your random forests classifier to improve classification, without actually running the classifier? Is there a screening process? How do you do it? :) –  Vishal Jul 25 '12 at 11:45
I don't know of any useful a-priori selection methods. You can do many nested loops of importance and selection via some R packages like Boruta. I haven't found them useful either. I find it unreasonable to believe any given feature has no effect. I can believe emphasizing certain features over others could be useful, but the base randomForest algorithm does this pretty well already. If you are that deep into modeling and want more performance, I'd suggest stacking other algorithms, some as some boosted trees, with your randomForest. –  Shea Parkes Jul 25 '12 at 12:42
You could a priori calculate some separability measures for your classes based on your features (Jeffries-Matusita distance, Divergence, etc). This could help you figure out in general which features help you distinguish between classes but because of the workings of RF it isn't easy to choose from here which features provide the best set for classification. One clear obstacle here is that RF finds variable interactions by itself. –  JEquihua Mar 22 at 18:53