Do I use the mean vector from my training set to center my testing set when dimension reducing for classification?

Question

Please let me know if this is the right place to ask this (or if any of my tags are wrong) or if I need to write this any differently.

Do I use the mean vector from my training set to center my testing set when dimension reducing for classification?

I am using the principal component analysis procedure to reduce the dimensions of the training set. I build the classifier. Then, before I classify the feature vectors from the test set, during the centering part of the dimension reduction, do I use the same mean vector from the training set, do I take the mean vector of the testing set and subtract that from the test set, or do I take the mean vector of the union of the training and test set and subtract that from the test set?

If the third option, does that mean I was also supposed to use the union of the training and testing set to center the training set as well? No, (for the sake of generalizing to other testing sets) right?

Also, even though I am pretty sure the answer will be the same as above, can you please let me know if the same is true for using the covariance matrix from the training set to get an eigenvector matrix and multiplying the inverse (transverse) of it times the test set to reduce it. Or, do we use the testing set or the union of the two to get the covariance and then eigenvector matrix to multiply times the testing set?

Answer 1

You should use the estimates from the training dataset to adjust both testing and training parts.

The reasoning is two fold.

One is to avoid some data-leakage where you infuse additional information about the actual center of the data from the test set, which you are not supposed to have.

Second reason is more practical. When you have a final model you want to use it to classify new observations. When a new observation comes you will not want to re-train your model. What you will do is use the model you have. Hence you will center and scale the new observation based on the means and covariance matrix obtained at the time the model was trained.

Simply put - the means and covariance matrix should be considered as being part of your model.

Answer 2

Yes, standardisation should also use training data. Otherwise, it'll be data leakage. It maybe slight, but sometimes effective. Think about all the operations you do as a single pipeline, $\mathcal P$:

• Step 1: Standardisation
• Step 2: Dimensionality Reduction
• Step 3: Classifier/Regression

But, all together it is a model, $\mathcal M$. All the validation and testing procedures you apply for a single model applies to this structure.

For the covariance matrix, the answer is the same. You'll use your training set.

Some libraries, such as sklearn has a good abstraction for this concept, namely Pipelines, where you put all the operations in a sequential manner, e.g. [StdScaler, PCA, SVC] and use a single fit function for the overall pipeline, making it act like a single unit/model.

TL;DR : Any information you use from the test set is basically data-leakage.