Why standardization of the testing set has to be performed with the mean and sd of the training set?

Question

In pre-processing the data set before applying a machine learning algorithm the data can be centered by subtracting the mean of the variable, and scaled by dividing by the standard deviation.

This is a straightforward process in the training set, but when it comes to the testing set, the procedure seems more ad hoc. I have read that the mean that is subtracted from each value in the testing set is the mean of the training set, not the testing set; and the same goes for the standard deviation.

Is there really a mathematical need behind this asymmetry, or is it an exercise in sticking to the principle of not touching the testing set until the end - more of a "philosophical" heuristic?

Answer 1

When you center and scale a variable in the training data using the mean and sd of that variable calculated on the training data, you are essentially creating a brand-new variable. Then you are doing, say, a regression on that brand new variable.

To use that new variable to predict for the validation and/or test datasets, you have to create the same variable in those data sets. Subtracting a different number and dividing by a different number does not create the same variable.

Answer 2

Let me explain a different way. Suppose you had distance measured in m. So X = distance in meters. But that is cumbersome, because some of the values of X are 50,000. So you create a new variable, X1 = distance in km. You obtain the values of X1 by dividing X by 1000. Now you build a model based on X1. You must also create X1 in your test data by dividing X by 1000. If you divide by 1001, or by 5,000, you aren't creating X1, you're creating X2, which has a completely different definition. Any model built based on X1 in the training data will not work if you use X2 in the test data instead of X1.

Now, centering and scaling is creating a new variable. You do not have to use the mean and sd of the training data. You could use the mean and sd of the whole dataset before splitting off into training vs. test. You could use the mean and sd of the test data. You could use a number kind of close to the mean and kind of close to the sd. You don't have to perfectly center and scale to create more stability in the design matrix. Bottom line is, if you create X1 = (X-5)/2.865, and then build a model with X1 as a predictor using training data, then if you create X2 = (X-5.375)/2 in your test data, and then act like it's the same new variable as X1, your model will not perform as well as it should, and it is an inappropriate use of the model.

Answer 3

Why should we use the mean and std of the training dataset to standardize the test dataset?

It is possible that the mean and std of the test dataset are such that after standardizing it with these values, some test data points will end up having same values as some (but different) train data points of the standardized train dataset (standardized by its own mean and std). See here for an example that demonstrates this.

Conversely, the test dataset could contain data points that are also contained in the train dataset, and if we standardize the ones that are in test dataset by the mean and std of the test dataset, and the ones that are in train dataset by the mean and std of the train dataset, they will end up having different values (assuming that the mean and std are different for train and test datasets).

Although this two situations are highly hypothetical, they demonstrate that by performing different transformations on two different data points can give same data point, whereas two same data points (one from train dataset and one from test dataset) can end up being different after the transformations.