13
$\begingroup$

For a recent Kaggle competition, I (manually) defined 10 additional features for my training set, which would then be used to train a random forests classifier. I decided to run PCA on the dataset with the new features, to see how they compared to each other. I found that ~98% of the variance was carried by the first component (the first eigenvector). I then trained the classifier multiple times, adding one feature at a time, and used cross-validation and RMS error to compare the quality of the classification. I found that the classifications improved with each additional feature, and that the final result (with all 10 new features) was far better than the first run with (say) 2 features.

  • Given that PCA claimed ~98% of the variance was in the first component of my dataset, why did the quality of the classifications improve so much?

  • Would this hold true for other classifiers? RF scales across multiple cores, so it's much faster to train than (say) SVM.

  • What if I had transformed the dataset into the "PCA" space, and run the classifier on the transformed space. How would my results change?

Improve this question
$\endgroup$
4
  • 2
    $\begingroup$ Did you normalize your data before running PCA? If I had to guess I'd think one of your features was on a much larger scale than the others... $\endgroup$
    – Marc Shivers
    Jul 23, 2012 at 14:08
  • 1
    $\begingroup$ The PCA function automatically normalizes everything when doing the calculation. $\endgroup$
    – Vishal
    Jul 23, 2012 at 14:16
  • 1
    $\begingroup$ Maybe that's just me, but could you clarify the following: your first step consisted in adding 10 (raw) features, one at a time, or did you work directly with PCA components? As stated, I understand this is the first case and you wonder whether you could work directly with results from PCA. In either case, did you apply the PCA on all variables, including new features, or just on the later? $\endgroup$
    – chl
    Jul 23, 2012 at 16:18
  • $\begingroup$ I applied the PCA to the original matrix with the 10 additional features. I then trained the classifier, by adding in one feature at a time, so I could measure the incremental improvement provided with the addition of each feature. My question was what if I transformed the dataset (with the 10 new features) into the PCA space, and then ran the classifier directly on the dataset in the PCA space $\endgroup$
    – Vishal
    Jul 23, 2012 at 17:16

2 Answers 2

Reset to default
8
$\begingroup$

When doing predictive modeling, you are trying to explain the variation in the response, not the variation in the features. There is no reason to believe that cramming as much of the feature variation into a single new feature will capture a large amount of the predictive power of the features as a whole.

This is often explained as the difference between Principal Component Regression instead of Partial Least Squares.

Improve this answer
$\endgroup$
5
  • $\begingroup$ "There is no reason to believe that cramming as much of the feature variation into a single new feature will capture a large amount of the predictive power of the features as a whole." That was never the point, and doing this would result in a very confused classifier! The goal was to have a range of features, all illustrating different aspects of the dataset, with the intention of reducing the generalization error. The point of taking the PCA was see how different the features were. And my point of posting was that my features were not that different, yet the results of RF still improved. $\endgroup$
    – Vishal
    Jul 24, 2012 at 14:54
  • 1
    $\begingroup$ The same logic can still apply. A new feature is highly colinnear with a prior feature can still bring more predictive power. Specifically for a randomForest: if the near duplicate feature is important in general, one version or another is now more likely to be selected as splitting candidates. $\endgroup$
    – Shea Parkes
    Jul 25, 2012 at 3:14
  • $\begingroup$ This begs the follow-up question, how do you a-priori select features for your random forests classifier to improve classification, without actually running the classifier? Is there a screening process? How do you do it? :) $\endgroup$
    – Vishal
    Jul 25, 2012 at 11:45
  • $\begingroup$ I don't know of any useful a-priori selection methods. You can do many nested loops of importance and selection via some R packages like Boruta. I haven't found them useful either. I find it unreasonable to believe any given feature has no effect. I can believe emphasizing certain features over others could be useful, but the base randomForest algorithm does this pretty well already. If you are that deep into modeling and want more performance, I'd suggest stacking other algorithms, some as some boosted trees, with your randomForest. $\endgroup$
    – Shea Parkes
    Jul 25, 2012 at 12:42
  • 1
    $\begingroup$ You could a priori calculate some separability measures for your classes based on your features (Jeffries-Matusita distance, Divergence, etc). This could help you figure out in general which features help you distinguish between classes but because of the workings of RF it isn't easy to choose from here which features provide the best set for classification. One clear obstacle here is that RF finds variable interactions by itself. $\endgroup$
    – JEquihua
    Mar 22, 2013 at 18:53
7
$\begingroup$

The first principal component is a linear combination of all your features. The fact that it explains almost all the variability just means that most of the coefficients of the variables in the first principal component are significant.

Now the classification trees you generate are a bit of a different animal too. They do binary splits on continuous variables that best separate the categories you want to classify. That is not exactly the same as finding orthogonal linear combinations of continuous variables that give the direction of greatest variance. In fact we have recently discussed a paper on CV where PCA was used for cluster analysis and the author(s) found that there are situations wher best separation is found not in the 1st few principal components but rather in the last ones.

Improve this answer
$\endgroup$
3
  • 3
    $\begingroup$ "In fact we have recently discussed a paper on CV where PCA" do you have a link to this? I'm very interested :) $\endgroup$
    – user603
    Jul 23, 2012 at 15:06
  • $\begingroup$ I will look for the discussion. $\endgroup$
    – Michael R. Chernick
    Jul 23, 2012 at 16:46
  • $\begingroup$ Will you be so kind to take a look at the related question? $\endgroup$
    – nadya
    Nov 7, 2012 at 0:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.