5
$\begingroup$

I'm doing an image classification task and the number of features of each example image is pretty huge (3,072: # pixels in each image). I'm thinking of using PCA to reduce the # features of each image to $n$ (say $n = 100$), and then use SVM to to learn and classify using the reduced feature space. I'm wondering which of these two paths should I follow?

  1. PCA on the training set $T$ and use SVM to learn on new $T$ with reduced dimensions. For prediction, PCA again on the test set $S$ and use the learned SVM parameters to classify.

  2. PCA on both $T$ and $S$ at the same time. That is, merge $T+S$ into a large matrix and perform PCA on them. Then split the reduced dimension matrix into $T$ and $S$ again. Learn the SVM on the reduced $T$ and then use the learned SVM to predict on the reduced $S$.

$\endgroup$
1
  • 4
    $\begingroup$ I would like to call your attention to the fact that katya's answer is currently marked as accepted by you, but is dangerously misleading and advocates something that should not be done. This is explained in Dima's answer (which also has more upvotes). Do you maybe want to reconsider and mark that one as accepted? Just so you know, one can always change which answer is marked as accepted. $\endgroup$
    – amoeba
    Dec 16 '14 at 12:53
1
$\begingroup$

In the context of this problem, (2) makes more sense, because otherwise you may not even have the same features you are trying to classify (ie reduced dimensions may mean very different things). See here for a more detailed discussion https://stackoverflow.com/questions/10818718/principal-component-analysis

$\endgroup$
4
  • $\begingroup$ I agree with you (2) makes more sense now. Do you think the same would apply if I replace SVM with something else such as ANN (artificial neural networks)? $\endgroup$
    – Joe
    Nov 23 '14 at 20:58
  • $\begingroup$ My comment refers to interpretability of PCs, not specific to any particular method applied to them, so I would think it still holds. Just imagine that otherwise you might be using a training set-PC1 which is RGB to classify test-PC1 which is eg degree of blur (forgive the poor example of one not familiar with your system). $\endgroup$
    – katya
    Nov 23 '14 at 21:12
  • 4
    $\begingroup$ (-1) This accepted answer unfortunately advocates something that should not be done, see @Dima's answer below. $\endgroup$
    – amoeba
    Dec 13 '14 at 23:15
  • $\begingroup$ This should not be the accepted answer, the one below should be! $\endgroup$
    – Dinesh
    Mar 22 '17 at 15:22
22
$\begingroup$

(1) is incorrect, because if you run PCA on the two sets separately, you will end up with two different spaces. You cannot train a classifier in one space, and apply it to a different space.

(2) is cheating. When you train a classifier, you cannot use any information from the test set.

The correct way would be to run PCA on the training set, save the principal components that you use, and then use them to transform the points in your test set. This way the points in both sets end up in the same space, and you are not using any knowledge about your test set during training.

Alternatively, you can use an entirely separate data set, just for computing the principal components. Then project both your training set and your test set into the space defined by those.

$\endgroup$
5
  • 1
    $\begingroup$ I wonder if it is truly cheating because you are re-defining your response variables to become PCs, then the regular, unbiased test-training process follows. For example, using the same response variables in test and training sets is not only justified, but expected, and same would hold for reduced-dimension data such as PCs. It is an interesting question, though, leaving room to consider both approaches. $\endgroup$
    – katya
    Nov 25 '14 at 1:28
  • 3
    $\begingroup$ @katya, it is cheating, because you are using information from the test set, which you are not supposed to be aware of during training. It may not have much of an effect, if the test set is much smaller than the training set, but, strictly speaking, it does not make for a clean experiment. $\endgroup$
    – Dima
    Nov 25 '14 at 13:13
  • 1
    $\begingroup$ This is a great answer. $\endgroup$
    – Mecasickle
    Sep 29 '15 at 20:15
  • 1
    $\begingroup$ The best answer ! $\endgroup$
    – Dinesh
    Mar 22 '17 at 15:21
  • $\begingroup$ The accepted answer should not be accepted. This one is right. You never peak at testing or validation data. A real machine learning product is used on data that did not even exist when it was created. For instance, Siri's speech recognition should be able to make sense of a baby's first words. That baby had not generated any previous speech data on which Siri could have trained. $\endgroup$
    – Dave
    Aug 7 '19 at 14:07

Not the answer you're looking for? Browse other questions tagged or ask your own question.