# Should standardization be done using leave-one-out?

When we have data from a normal distribution, we may wish to standardize the values in our sample to $$N(0,1)$$. In such case it is customary to subtract the sample mean from each observation and then divide by the standard deviation (sd). However, shouldn't this be done using a leave-one-out calculation of the mean and sd for each data point? That is, for each observation we would remove it from the data, calculate the mean and sd, use them to standardize the observation - and repeat this process for all the data points.

This alternative solution is obviously computationally intensive, but does it give "better" results?

Answers (and references) are very welcome.

• There are many different reasons one may want to standardize some variable. Is your intuition that using values based on LOO should be "better," in some sense, no matter what reason you're standardizing for? Jun 21, 2019 at 3:53
• Leave one out would not change the mean and standard deviation calculated(in the final model) , just allow you to explore its certainty. If you want to Winsorize your values before using for standardisation, it would be a way to get there. Jun 21, 2019 at 4:46
• @JakeWestfall, for example, if I wanted to do a qq-plot of the standardized values with the standard normal distribution (N(0,1)). Would it be better (i.e.: less biased) estimation of the quantile of each value, if I calculate it using loo? Jun 24, 2019 at 13:49
• One issue behind this question is that if you have n data points iid from some normal distribution and you standardise them in the standard way, the resulting data will no longer be independent (because their sample mean has to be 0 and their sample variance has to be 1). They will therefore not behave like an iid sample from N(0,1) (I'm not quite sure at first sight whether at least the marginals will be N(0,1); otherwise it would be misleading to say we "standardise to N(0,1)). Unfortunately this problem doesn't seem to be solved by LOO-standardisation either. Jun 26, 2019 at 15:57

If the goal is to standardize the data set (mean center and measure distance from the mean in standard deviation units), then the leave-one-out (LOO) approach simply is not correct. This can be best seen with a simple data set (with an outlier, to exaggerate the point). $$\{47.5,50.7,55.7,58,42.1,51.8,40.8,39.9,45.6,95\}$$ If you convert to standardized ($$z$$) scores, then you obtain a new data set with a mean of zero and a standard deviation of one. If, on the other hand, you Studentize (standardize with LOO) the scores, you obtain a data set with a mean of 0.437±2.417. Thus, you have failed to "standardize" the data set.

Now, if there is a broader goal that you are attempting to achieve via standardizing the data set, then it may be the case that you want to studentize. For example, if you wish to assess to what degree the outlier in the data set above is actually "not following the pattern" of the rest of the data, the standardized value is $$z_{(10)}=2.63$$ but the studentized value is $$z_{(10)}=7.22$$.

Note, one might argue that the outlier is so extreme that either approach should flag it. However, if you have multiple outliers, this may not always be the case (particularly if they occur on both ends of the distribution). To see this, you can change the first value to $$x_{(1)} = 97.5$$ to see how you might miss outliers with standardized scores. You can also change the first value to $$x_{(1)}=7.5$$ to see how you might be able to detect more extreme trends away from "the rest of the data".

• Why do you call it "studentize"? This is not what it means: en.wikipedia.org/wiki/Studentization Jun 23, 2019 at 19:46
• Please check the link at the bottom of the page for studentized residual...the deviation score is indeed a residual (for a null model). Jun 23, 2019 at 21:11
• en.wikipedia.org/wiki/Studentized_residual Thanks Gregg. What I'm wondering to myself is if you have some references talking about this more in depth and showing examples. Jun 24, 2019 at 14:19
• sorry about references...I pretty much went off the top of my head for my answer...there might be something in Faraway's book on linear models for R (I recall studentized residuals there, but that would be in a regression context, not a descriptive context). Lomax & van Hahs-Vaughn may have something (but it will be in the context of multiple comparison protocols...eg studentized ranges). The only other ref I'd think of checking is Tukey's original work about EDA (circa 1970s, but can't guarantee this would be mentioned there) Jun 24, 2019 at 17:58

Given that you used the term leave-one-out, I think what is implied by the question is that you want to do a LOO cross-validation. That is, you want to train on n-1 examples and test your model on the remaining example. In that case, being very methodologically strict, you are correct. The standardization is performed only on the training set, in this case, the n-1 examples. The for the test example, you subtract the mean of the training set and divide by the standard deviation of the training set. This will true for any cross-validation you choose. Standardization is part of the model you are creating, and you learn the standardization using only the training set, as you learn any other parameters using only the training set. And you apply the model on the test set.

In practice, almost never you need to be so strict. You can learn the standardization parameters (mean and standard deviation) on the whole set, and only then use the cross-validation. What is the consequence of doing this? What is happening is that once you define the cross-validation, you are using some information of the test set (its contribution to the general mean and standard deviation) in the training. That will usually result in a cross-validation measurement that is somewhat optimistic. If you are measuring X and the higher the X the better, using some information of the test set in learning step will result in a higher X measured on the test set that the "true" one.

If your final goal is to report the X itself, then your measurement will be a little bit inflated. This will be worse if you have few data points. But if your goal is to use the measure X to select among different models, then most likely the inflated X will be "the same" for all the models, and the order will be preserved, or, at least, most likely the best model will be the same in both cases. Again this will be more likely the more data you have.

To summarize, if you have a lot of data, do not worry, perform the standardization on the whole data set. If you have little data (and that will be a good reason to use LOO instead of other cross-validation schemes) then do perform the standardization as you described.