6
$\begingroup$

I have train and test data set to work on. I would like to apply PCA to reduce dimension.

Do I need to merge train and test data sets together before applying PCA? Or I should apply PCA on train data set, select reduced dimension, and work on reduced data set for both train and test data set?

Improve this question
$\endgroup$
0

2 Answers 2

Reset to default
6
$\begingroup$

Principal component analysis will provide you with a number of principal components $W$; these components will qualitatively represent the principal and orthogonal modes of variation in your sample. You will use (some) of these $W$ to project your original dataset $X$ to a lower dimensional subspace $T$. This is your new dataset and the PCs are in effect an axis system over which we can represent data $X$ in a compact form.

Now, as @RobertKubrick mentions you need to make sure that information from your testing dataset is not "leaked" into your training dataset. If this takes place then you will utilize information that "should be unknown" during your prediction; your error estimates will be wrong. The generalization of your model will suffer.

For your case in particular you should do the following: Calculate the principal components $W$s on the training dataset and then utilize the training sample $W$ to reduce the dimensions of the testing dataset. I say this because:

  1. if you merged both your training and testing dataset to calculate your PC you will evidently utilize information from the testing set. This is clearly wrong.
  2. If you did two independent PCAs you will be comparing data registered on different axes (if anything princ. components are not sign-identifiable so the estimated parameters from them will also have the same issue). The axes over which you project your data should be the same, otherwise you are in a typical "orange-apples situation".

Clearly if you do $k$-fold cross validation, or something similar (eg. jack-knifing) that you will need to calculate new principal components $W$ each time. I.T. Jolliffe's Principal Component Analysis is a standard and great reference on PCA; I would strongly recommend it.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thanks for your reply. Are you suggesting to go for k-fold cross validation on training data set? I do cross validation on training data all time, but never reduced dimension using PCA. Not sure I do need k-fold cross validation at PCA analysis or not. So, this is bit confusing for me. Would you please guide me on it? $\endgroup$
    – user3037153
    Commented Sep 6, 2014 at 23:52
  • $\begingroup$ Sorry, I did not mean to imply that you do cross-validation for the PCA itself. My comment for recalculating PCA referred to what you would need to do within each of the $k$ iterations. $\endgroup$
    – usεr11852
    Commented Sep 7, 2014 at 0:25
3
$\begingroup$

The test set should never be included in your modeling decisions or else you will be lose the benefit of unfitted data. This is true for regression, PCA or whatever other fitting technique.

You want to calculate the prediction error on data "unseen" by your model.

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ @Robert Let's say your dataset has 500 dimensions, you run PCA on the training set to reduce to 50D, run a classifier on that and get good results. Now how do you use this classifier on the test data? Do you reduce the test data dimensions to 50 via another PCA and use the classifier? $\endgroup$
    – gokul_uf
    Commented Oct 26, 2016 at 13:03

Not the answer you're looking for? Browse other questions tagged or ask your own question.