34
$\begingroup$

I have a dataset for which I have multiple sets of binary labels. For each set of labels, I train a classifier, evaluating it by cross-validation. I want to reduce dimensionality using principal component analysis (PCA). My question is:

Is it possible to do the PCA once for the whole dataset and then use the new dataset of lower dimensionality for cross-validation as described above? Or do I need to do a separate PCA for every training set (which would mean doing a separate PCA for every classifier and for every cross-validation fold)?

On one hand, the PCA does not make any use of the labels. On the other hand, it does use the test data to do the transformation, so I am afraid it could bias the results.

I should mention that in addition to saving me some work, doing the PCA once on the whole dataset would allow me to visualize the dataset for all label sets at once. If I have a different PCA for each label set, I would need to visualize each label set separately.

$\endgroup$
33
$\begingroup$

For measuring the generalization error, you need to do the latter: a separate PCA for every training set (which would mean doing a separate PCA for every classifier and for every CV fold).

You then apply the same transformation to the test set: i.e. you do not do a separate PCA on the test set! You subtract the mean (and if needed divide by the standard deviation) of the training set, as explained here: Zero-centering the testing set after PCA on the training set. Then you project the data onto the PCs of the training set.


  • You'll need to define an automatic criterium for the number of PCs to use.
    As it is just a first data reduction step before the "actual" classification, using a few too many PCs will likely not hurt the performance. If you have an expectation how many PCs would be good from experience, you can maybe just use that.

  • You can also test afterwards whether redoing the PCA for every surrogate model was necessary (repeating the analysis with only one PCA model). I think the result of this test is worth reporting.

  • I once measured the bias of not repeating the PCA, and found that with my spectroscopic classification data, I detected only half of the generalization error rate when not redoing the PCA for every surrogate model.

That being said, you can build an additional PCA model of the whole data set for descriptive (e.g. visualization) purposes. Just make sure you keep the two approaches separate from each other.


I am still finding it difficult to get a feeling of how an initial PCA on the whole dataset would bias the results without seeing the class labels.

But it does see the data. And if the between-class variance is large compared to the within-class variance, between-class variance will influence the PCA projection. Usually the PCA step is done because you need to stabilize the classification. That is, in a situation where additional cases do influence the model.

If between-class variance is small, this bias won't be much, but in that case neither would PCA help for the classification: the PCA projection then cannot help emphasizing the separation between the classes.

$\endgroup$
  • $\begingroup$ Thanks, this is exactly what I thought so it is good to hear it from an independent source. I am still finding it difficult to get a feeling of how an initial PCA on the whole dataset would bias the results without seeing the class labels. $\endgroup$ – Bitwise Apr 10 '13 at 18:10
  • $\begingroup$ @Bitwise: please see my edit $\endgroup$ – cbeleites Apr 10 '13 at 20:59
  • $\begingroup$ Hi @cbeleites, I want to make this thread a "canonical" thread for the questions about PCA and train/test splitting (there are many!) and mark those as duplicates. I took the liberty to add one sentence to your answer that might clear up a misunderstanding that often arises in the duplicate questions. Hope you are happy with my edit, but please check! +1, btw. $\endgroup$ – amoeba Dec 16 '14 at 21:53
  • $\begingroup$ @amoeba, thank you very much. Yes, that is an important point you added. Any many thanks for the work you put into cleaning up the collection of questions. $\endgroup$ – cbeleites Dec 17 '14 at 10:13
  • 1
    $\begingroup$ @FelipeAlmeida: yes $\endgroup$ – cbeleites Jul 19 '16 at 8:13
2
$\begingroup$

The answer to this question depends on your experimental design. PCA can be done on the whole data set so long as you don't need to build your model in advance of knowing the data you are trying to predict. If you have a dataset where you have a bunch of samples some of which are known and some are unknown and you want to predict the unknowns, including the unknowns in the PCA will give you are richer view of data diversity and can help improve the performance of the model. Since PCA is unsupervised, it isn't "peaking" because you can do the same thing to the unknown samples as you can to the known.

If, on the other hand, you have a data set where you have to build the model now and at some point in the future you will get new samples that you have to predict using that prebuilt model, you must do separate PCA in each fold to be sure it will generalize. Since in this case we won't know what the new features might look like and we can't rebuild the model to account for the new features, doing PCA on the testing data would be "peaking". In this case, both the features and the outcomes for the unknown samples are not available when the model would be used in practice, so they should not be available when training the model.

$\endgroup$
0
$\begingroup$

Do the latter, PCA on training set each time

In PCA, we learn the reduced matrix : U which helps us get the projection Z_train = U x X_train

At test time, we use the same U learned from the training phase and then compute the projection Z_test = U x X_test

So, essentially we are projecting the test set onto the reduced feature space obtained during the training.

The underlying assumption, is that the test and train set should come from the same distribution, which explains the method above.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.