"We have to apply feature scaling on test set using scaling parameters from train set." Is this statement true? If yes, why?

Question

I understand that in standard data cleaning and pre-processing pipelines, we have to make sure that the information from the test set (or what would be the test set after splitting) does not leak into the training process, so that it simulates real-world scenarios where real test data, for which we are building the model, is unknown at the time of training the model.

However, in processes like scaling the features, such as min-max-scaling, $x'={\frac {x-{\text{min}}(x)}{{\text{max}}(x)-{\text{min}}(x)}}$ and standardisation, $x' = \frac{x - \bar{x}}{\sigma}$, why should we use the scaling parameters (min, max, mean, variance) from the training set when scaling the test set? Why can't we compute these parameters from the test set itself and use them to scale test set? As for as I can tell, no information is being leaked or assumed. (I know ideally both sets should have similar values for these parameters.)

The second (worrying) part of the question is, by scaling the test set with its own parameters, have I only been compromising the performance of the model or have I been breaking some more fundamental logic behind machine learning process?

Answer 1

I think you've answered your own question -- the scale of truly out-of-sample data that we might see "in the wild" is unknown. The reason that the out-of-sample data is unknown is quite obvious: we simply didn't collect the data. The best that we can do is to center and scale the data using the information from the training set. The purpose of using the training data to derive the scale information, instead of the testing data, is to simulate the out-of-sample data.

If you don't maintain this partition and instead you use centering and scaling derived from the test data, this is a distortion of the process in the sense that you are no longer simulating the out-of-sample properties of your method, so whatever information you were attempting to derive by using a train-test partition has been compromised because the partitioning was violated. What that means in specific terms depends on the specifics of the model and the data.

Consider an example of linear regression.

If you change the centering and scaling parameters to reflect new data, then you're also implicitly changing the underlying model since $x$ and $x^\prime$ are different, so too are all parts of the model that depend on the scaled data. Suppose the model is linear regression trained on data with mean 5 and standard deviation 2. All of the coefficients will depend on this centering and scaling. Any other centering/scaling values will be indirectly adjusting the coefficients and therefore changing the model's predictions.