PCA on train and test datasets: should I run one PCA on train+test or two separate on train and on test? I'm doing an image classification task and the number of features of each example image is pretty huge (3,072: # pixels in each image). I'm thinking of using PCA to reduce the # features of each image to $n$ (say $n = 100$), and then use SVM to to learn and classify using the reduced feature space. I'm wondering which of these two paths should I follow?


*

*PCA on the training set $T$ and use SVM to learn on new $T$ with reduced dimensions. For prediction, PCA again on the test set $S$ and use the learned SVM parameters to classify.

*PCA on both $T$ and $S$ at the same time. That is, merge $T+S$ into a large matrix and perform PCA on them. Then split the reduced dimension matrix into $T$ and $S$ again. Learn the SVM on the reduced $T$ and then use the learned SVM to predict on the reduced $S$.
 A: (1) is incorrect, because if you run PCA on the two sets separately, you will end up with two different spaces. You cannot train a classifier in one space, and apply it to a different space.
(2) is cheating. When you train a classifier, you cannot use any information from the test set.
The correct way would be to run PCA on the training set, save the principal components that you use, and then use them to transform the points in your test set.  This way the points in both sets end up in the same space, and you are not using any knowledge about your test set during training.
Alternatively, you can use an entirely separate data set, just for computing the principal components.  Then project both your training set and your test set into the space defined by those.
A: In the context of this problem, (2) makes more sense, because otherwise you may not even have the same features you are trying to classify (ie reduced dimensions may mean very different things). See here for a more detailed discussion https://stackoverflow.com/questions/10818718/principal-component-analysis
