6
$\begingroup$

In pre-processing the data set before applying a machine learning algorithm the data can be centered by subtracting the mean of the variable, and scaled by dividing by the standard deviation.

This is a straightforward process in the training set, but when it comes to the testing set, the procedure seems more ad hoc. I have read that the mean that is subtracted from each value in the testing set is the mean of the training set, not the testing set; and the same goes for the standard deviation.

Is there really a mathematical need behind this asymmetry, or is it an exercise in sticking to the principle of not touching the testing set until the end - more of a "philosophical" heuristic?

$\endgroup$
  • $\begingroup$ There is a great example in the final chapter of "learning from data" illustrating the necessity of this. $\endgroup$ – Matthew Drury Mar 18 '16 at 1:12
6
$\begingroup$

When you center and scale a variable in the training data using the mean and sd of that variable calculated on the training data, you are essentially creating a brand-new variable. Then you are doing, say, a regression on that brand new variable.

To use that new variable to predict for the validation and/or test datasets, you have to create the same variable in those data sets. Subtracting a different number and dividing by a different number does not create the same variable.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ That makes sense, but I'm just starting to look into ML, and isn't the final goal to have an algorithm for the initial variable beyond the actual sample (data set) you are working with... Extrapolate... $\endgroup$ – Antoni Parellada Mar 18 '16 at 1:35
5
$\begingroup$

Let me explain a different way. Suppose you had distance measured in m. So X = distance in meters. But that is cumbersome, because some of the values of X are 50,000. So you create a new variable, X1 = distance in km. You obtain the values of X1 by dividing X by 1000. Now you build a model based on X1. You must also create X1 in your test data by dividing X by 1000. If you divide by 1001, or by 5,000, you aren't creating X1, you're creating X2, which has a completely different definition. Any model built based on X1 in the training data will not work if you use X2 in the test data instead of X1.

Now, centering and scaling is creating a new variable. You do not have to use the mean and sd of the training data. You could use the mean and sd of the whole dataset before splitting off into training vs. test. You could use the mean and sd of the test data. You could use a number kind of close to the mean and kind of close to the sd. You don't have to perfectly center and scale to create more stability in the design matrix. Bottom line is, if you create X1 = (X-5)/2.865, and then build a model with X1 as a predictor using training data, then if you create X2 = (X-5.375)/2 in your test data, and then act like it's the same new variable as X1, your model will not perform as well as it should, and it is an inappropriate use of the model.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ (+1) Good explanation, but since you're relatively new here, @Amy: I don't think this is really a situation where a second answer is needed, since this is basically just an addition to your other post. It probably would have been cleaner to edit that post to add this in. (No need to do that now, just for the future.) $\endgroup$ – djs Mar 18 '16 at 4:25
1
$\begingroup$

Why should we use the mean and std of the training dataset to standardize the test dataset?

It is possible to have a situation where the test dataset's mean and std are such that some (or all) test samples of the standardized test dataset will end up having same values as some different samples of the standardized train dataset. See here for an example that demonstrates this.

Another possibility is for our test dataset to contain samples that are also contained in the train dataset, and if we standardize them by the mean and std of the test dataset, they will have different values from if they were standardized my mean and std of train dataset (assuming that the mean and std are different for both datasets), and this can lead to a different classification of the same data during training and evaluation.

So we should transform our test data in the same way as we transform the train data to avoid such situations.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.