# Why do we normalize test data on the parameters of the training data?

I just built a toy linear regression model with gradient descent, coding it from scratch. It was doing fine on test data, but it was off on training data. In the end I figured that I was normalizing new data according to its own mean and range, instead of using the mean and range of the training data.

And I realized that I never understood why this doesn't work, and I never found an explainer. Intuitively, I see normalization as a way of "rewriting" the data, without changing its structure. When I get the test data, I can easily calculate its mean and range and standard deviation. So shouldn't I "rewrite" it in terms of itself? Doing it with the statistics of the training data also feels a bit like cheating, since we typically avoid any contact between training set and testing set.

• Training data and test data are centered and scaled according to the training data because we don't want to encode information about the test data in the training data. This is what you write in your last sentence. Can you help me understand what's not clear to you?
– Sycorax
Commented Nov 6, 2020 at 15:36
• My intuition says that training data should be centered on training data and test data should be centered on test data. Centering test data on training data feels like we're encoding information about the training data in the test data. Commented Nov 6, 2020 at 15:39
• The purpose of a test data set is to simulate the effect of using the model in the future. You won't know the mean or standard deviation of that data because you don't have it. So, as a part of this simulation, we use the training data as a proxy/ansatz for that future data.
– Sycorax
Commented Nov 6, 2020 at 15:42
• In my toy problem, the "training data" included the integers in the range [0, 100]. And the "test data" included the integers in the range [100, 200]. So the vectors were similar, and for both of them it was easy to compute mean and range. Hence I was surprised when my algorithm failed if I didn't use the parameters of the first vector. It doesn't seem that I would need a proxy in this case. Why its actual own parameters are worse than the parameters of the training data is what I'm trying to understand. Commented Nov 6, 2020 at 15:46
• The assumption in the usual train/test split setup is that the training and testing data are iid realizations from the same distribution. You've used 2 different distributions for your training and your test data. It seems perfectly logical that this setup wouldn't work: the mean of the training data is 50, but the mean of the test data is 150.
– Sycorax
Commented Nov 6, 2020 at 15:48

## 2 Answers

You are supposed to use the parameters from the training data set to standardize the test data.

This is because in real life we only ever have access to some subset of the total population of data. When we deploy a data-driven (statistical) model it needs to adapt to new, unseen data. Since we do not have access to this new, unseen data we cannot realistically have some large "group" of it to do a separate standerdization on it. The test data is meant to simulate this part of our reality somewhat - there some data out there that we don't have access to, and eventually our model needs to process it based on "what it knows".

Sure perhaps we can always "wait" for more data to arrive, and re-train models + update statistics, but that is often expensive. Therefore we usually train once (in a long while), depending on what data we have available at the start, and try to estimate generalization error and our model's adaptability by ensuring we don't play around at all with test data during model building and data pre-processing.

Also as brought up in a comment above - it is important that the training and test come from (are reflective of) the same distribution - this is an assumption we place in our modelling.

Here is a simple example of why you should always standardize with the training mean and standard deviation.

You have training data $$x$$ that you standardize. The mean is 2 and the standard deviation is 1. When you get an observation of 3, the standardized value is 1.

Some time later, the means of the predictors shift for natural reasons. The mean is now 4 and the standard deviation is 1. When you get an observation of 3, the standardized value is -1.

If you were to use standardized test data which used the test sets means and standard deviations, your model would think that the observation of 3 in the test data corresponds to an observation of 1 in the training data. That is going to lead to a very poor prediction in some cases.

• Does anything change to your answer when If I would use Welford's online algorithm for getting actualized mean and standard deviation for new data? Commented Nov 7, 2021 at 18:00