3
$\begingroup$

I understand that in standard data cleaning and pre-processing pipelines, we have to make sure that the information from the test set (or what would be the test set after splitting) does not leak into the training process, so that it simulates real-world scenarios where real test data, for which we are building the model, is unknown at the time of training the model.

However, in processes like scaling the features, such as min-max-scaling, $x'={\frac {x-{\text{min}}(x)}{{\text{max}}(x)-{\text{min}}(x)}}$ and standardisation, $x' = \frac{x - \bar{x}}{\sigma}$, why should we use the scaling parameters (min, max, mean, variance) from the training set when scaling the test set? Why can't we compute these parameters from the test set itself and use them to scale test set? As for as I can tell, no information is being leaked or assumed. (I know ideally both sets should have similar values for these parameters.)

The second (worrying) part of the question is, by scaling the test set with its own parameters, have I only been compromising the performance of the model or have I been breaking some more fundamental logic behind machine learning process?

$\endgroup$

1 Answer 1

5
$\begingroup$

I think you've answered your own question -- the scale of truly out-of-sample data that we might see "in the wild" is unknown. The reason that the out-of-sample data is unknown is quite obvious: we simply didn't collect the data. The best that we can do is to center and scale the data using the information from the training set. The purpose of using the training data to derive the scale information, instead of the testing data, is to simulate the out-of-sample data.

If you don't maintain this partition and instead you use centering and scaling derived from the test data, this is a distortion of the process in the sense that you are no longer simulating the out-of-sample properties of your method, so whatever information you were attempting to derive by using a train-test partition has been compromised because the partitioning was violated. What that means in specific terms depends on the specifics of the model and the data.

Consider an example of linear regression.

If you change the centering and scaling parameters to reflect new data, then you're also implicitly changing the underlying model since $x$ and $x^\prime$ are different, so too are all parts of the model that depend on the scaled data. Suppose the model is linear regression trained on data with mean 5 and standard deviation 2. All of the coefficients will depend on this centering and scaling. Any other centering/scaling values will be indirectly adjusting the coefficients and therefore changing the model's predictions.

$\endgroup$
3
  • $\begingroup$ Thank you for the answer. The scale of unknown samples is unknown until these samples have arrived. But as soon as we have these samples in our pipeline, we can now compute their mean, variance etc and scale them as they come, even in the production setting. The process is still confined to within unknown samples; it is just analogous to what we did to train samples. I am sorry but I still don't see how this violates the spirit of train-test split. $\endgroup$ Aug 26, 2018 at 2:58
  • $\begingroup$ If you change the centering and scaling parameters to reflect new data, then you're also implicitly changing the underlying model since $x$ and $x^\prime$ are different, so too are all parts of the model that depend on the scaled data. Suppose the model is linear regression trained on data with mean 5 and standard deviation 2. All of the coefficients will depend on this scaling. Any other scaling will be indirectly adjusting the coefficients and therefore changing the model's predictions. $\endgroup$
    – Sycorax
    Aug 26, 2018 at 3:11
  • $\begingroup$ You should also notice that you may get the new samples one at a time instead of in batch, and you still need to be able to score the model. $\endgroup$ Aug 26, 2018 at 5:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.