Should standardization be done using leave-one-out?

Question

When we have data from a normal distribution, we may wish to standardize the values in our sample to $N(0,1)$. In such case it is customary to subtract the sample mean from each observation and then divide by the standard deviation (sd). However, shouldn't this be done using a leave-one-out calculation of the mean and sd for each data point? That is, for each observation we would remove it from the data, calculate the mean and sd, use them to standardize the observation - and repeat this process for all the data points.

This alternative solution is obviously computationally intensive, but does it give "better" results?

Answers (and references) are very welcome.

Answer 1

If the goal is to standardize the data set (mean center and measure distance from the mean in standard deviation units), then the leave-one-out (LOO) approach simply is not correct. This can be best seen with a simple data set (with an outlier, to exaggerate the point). $$\{47.5,50.7,55.7,58,42.1,51.8,40.8,39.9,45.6,95\}$$ If you convert to standardized ($z$) scores, then you obtain a new data set with a mean of zero and a standard deviation of one. If, on the other hand, you Studentize (standardize with LOO) the scores, you obtain a data set with a mean of 0.437±2.417. Thus, you have failed to "standardize" the data set.

Now, if there is a broader goal that you are attempting to achieve via standardizing the data set, then it may be the case that you want to studentize. For example, if you wish to assess to what degree the outlier in the data set above is actually "not following the pattern" of the rest of the data, the standardized value is $z_{(10)}=2.63$ but the studentized value is $z_{(10)}=7.22$.

Note, one might argue that the outlier is so extreme that either approach should flag it. However, if you have multiple outliers, this may not always be the case (particularly if they occur on both ends of the distribution). To see this, you can change the first value to $x_{(1)} = 97.5$ to see how you might miss outliers with standardized scores. You can also change the first value to $x_{(1)}=7.5$ to see how you might be able to detect more extreme trends away from "the rest of the data".

Answer 2

Given that you used the term leave-one-out, I think what is implied by the question is that you want to do a LOO cross-validation. That is, you want to train on n-1 examples and test your model on the remaining example. In that case, being very methodologically strict, you are correct. The standardization is performed only on the training set, in this case, the n-1 examples. The for the test example, you subtract the mean of the training set and divide by the standard deviation of the training set. This will true for any cross-validation you choose. Standardization is part of the model you are creating, and you learn the standardization using only the training set, as you learn any other parameters using only the training set. And you apply the model on the test set.

In practice, almost never you need to be so strict. You can learn the standardization parameters (mean and standard deviation) on the whole set, and only then use the cross-validation. What is the consequence of doing this? What is happening is that once you define the cross-validation, you are using some information of the test set (its contribution to the general mean and standard deviation) in the training. That will usually result in a cross-validation measurement that is somewhat optimistic. If you are measuring X and the higher the X the better, using some information of the test set in learning step will result in a higher X measured on the test set that the "true" one.

If your final goal is to report the X itself, then your measurement will be a little bit inflated. This will be worse if you have few data points. But if your goal is to use the measure X to select among different models, then most likely the inflated X will be "the same" for all the models, and the order will be preserved, or, at least, most likely the best model will be the same in both cases. Again this will be more likely the more data you have.

To summarize, if you have a lot of data, do not worry, perform the standardization on the whole data set. If you have little data (and that will be a good reason to use LOO instead of other cross-validation schemes) then do perform the standardization as you described.