5
$\begingroup$

I'm doing an image classification task and the number of features of each example image is pretty huge (3,072: # pixels in each image). I'm thinking of using PCA to reduce the # features of each image to $n$ (say $n = 100$), and then use SVM to to learn and classify using the reduced feature space. I'm wondering which of these two paths should I follow?

  1. PCA on the training set $T$ and use SVM to learn on new $T$ with reduced dimensions. For prediction, PCA again on the test set $S$ and use the learned SVM parameters to classify.

  2. PCA on both $T$ and $S$ at the same time. That is, merge $T+S$ into a large matrix and perform PCA on them. Then split the reduced dimension matrix into $T$ and $S$ again. Learn the SVM on the reduced $T$ and then use the learned SVM to predict on the reduced $S$.

Improve this question
$\endgroup$
1
  • 4
    $\begingroup$ I would like to call your attention to the fact that katya's answer is currently marked as accepted by you, but is dangerously misleading and advocates something that should not be done. This is explained in Dima's answer (which also has more upvotes). Do you maybe want to reconsider and mark that one as accepted? Just so you know, one can always change which answer is marked as accepted. $\endgroup$
    – amoeba
    Commented Dec 16, 2014 at 12:53

2 Answers 2

Reset to default
1
$\begingroup$

In the context of this problem, (2) makes more sense, because otherwise you may not even have the same features you are trying to classify (ie reduced dimensions may mean very different things). See here for a more detailed discussion https://stackoverflow.com/questions/10818718/principal-component-analysis

Improve this answer
$\endgroup$
2
  • $\begingroup$ I agree with you (2) makes more sense now. Do you think the same would apply if I replace SVM with something else such as ANN (artificial neural networks)? $\endgroup$
    – Joe
    Commented Nov 23, 2014 at 20:58
  • $\begingroup$ My comment refers to interpretability of PCs, not specific to any particular method applied to them, so I would think it still holds. Just imagine that otherwise you might be using a training set-PC1 which is RGB to classify test-PC1 which is eg degree of blur (forgive the poor example of one not familiar with your system). $\endgroup$
    – katya
    Commented Nov 23, 2014 at 21:12
23
$\begingroup$

(1) is incorrect, because if you run PCA on the two sets separately, you will end up with two different spaces. You cannot train a classifier in one space, and apply it to a different space.

(2) is cheating. When you train a classifier, you cannot use any information from the test set.

The correct way would be to run PCA on the training set, save the principal components that you use, and then use them to transform the points in your test set. This way the points in both sets end up in the same space, and you are not using any knowledge about your test set during training.

Alternatively, you can use an entirely separate data set, just for computing the principal components. Then project both your training set and your test set into the space defined by those.

Improve this answer
$\endgroup$
5
  • 1
    $\begingroup$ I wonder if it is truly cheating because you are re-defining your response variables to become PCs, then the regular, unbiased test-training process follows. For example, using the same response variables in test and training sets is not only justified, but expected, and same would hold for reduced-dimension data such as PCs. It is an interesting question, though, leaving room to consider both approaches. $\endgroup$
    – katya
    Commented Nov 25, 2014 at 1:28
  • 3
    $\begingroup$ @katya, it is cheating, because you are using information from the test set, which you are not supposed to be aware of during training. It may not have much of an effect, if the test set is much smaller than the training set, but, strictly speaking, it does not make for a clean experiment. $\endgroup$
    – Dima
    Commented Nov 25, 2014 at 13:13
  • 1
    $\begingroup$ This is a great answer. $\endgroup$
    – Mecasickle
    Commented Sep 29, 2015 at 20:15
  • 1
    $\begingroup$ The best answer ! $\endgroup$
    – DVG
    Commented Mar 22, 2017 at 15:21
  • $\begingroup$ The accepted answer should not be accepted. This one is right. You never peak at testing or validation data. A real machine learning product is used on data that did not even exist when it was created. For instance, Siri's speech recognition should be able to make sense of a baby's first words. That baby had not generated any previous speech data on which Siri could have trained. $\endgroup$
    – Dave
    Commented Aug 7, 2019 at 14:07

Not the answer you're looking for? Browse other questions tagged or ask your own question.