2
$\begingroup$

I've come across the concept of data leakage in which optimistically biased generalisation errors occur due to test data in some sense 'seeing' the training data. For instance, normalisation on an entire dataset before train/test splitting occurs means the test set was normalised using parameters influenced by the training set.

I have been convinced, by this blog, to avoid it by performing train/test splits first, transforming the train data and then transforming the test data using parameters from the train set.

Some questions:

What is the advantage of using train parameters to transform the test set? Is it simply that the train set was larger and so a better estimate of the population parameters? What would be the consequences of using the test parameters to transform the test set - it avoids leakage, but perhaps has worse estimates for parameters and so performs more poorly?

I have an application (vibrational spectroscopy) in which some transformations (e.g. normalisation) occur at the per sample level (e.g. using the mean/std of the sample rather than the global). In this case I believe data leakage is not a concern, correct?

Does anyone know any published papers that explicitly look at data leakage and the effect it has on generalisation error? This is the best one I found, but couldn't find anything more.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

... test data in some sense 'seeing' the training data ...

The situation in data leakage is actually training data seeing test data, such that the test data leaks into your modelling process and affects the performance calculations. When you use training set statistics (e.g. mean/std) while standardising the test set, it's not called data leakage. Training set is always there since the model is dependent on that.

What is the advantage of using train parameters to transform the test set?

Normalisation/standardisation is also a part of the modelling process, and it should be completed before testing. Imagine your model is deployed and making predictions online (e.g. on user devices), you won't have access to aggregate test statistics because the test set is distributed across many devices online, so you use training statistics. The purpose of the test set is to emulate this behaviour so that you can have an idea of how your model will perform when it's deployed. Using test set statistics in your evaluation wouldn't be fair since you won't be able to do that in runtime.

Note that even if you have the test set (in runtime), you should be using the training set stats because otherwise would mean interfering with the model. Above mentioned scenario is just a good example.

In your particular case where the standardisation takes place per sample, it is domain specific, and not related to the data leakage mentioned above.

Improve this answer
$\endgroup$
2
  • $\begingroup$ I think the focus given to on-line prediction here is a red herring. You should normalize test data according to the training normalization parameters even in cases where you have test batches or a even a single, fixed test dataset that's entirely available at the time of prediction. Independent test set normalization is always undesirable, so the fact that it's in some cases impossible doesn't provide any additional justification. Even when you can, you shouldn't. $\endgroup$
    – Nuclear Hoagie
    Commented May 3, 2021 at 19:46
  • $\begingroup$ @NuclearHoagie thank you very much for your comment. Indeed, that was not my intention. $\endgroup$
    – gunes
    Commented May 3, 2021 at 19:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.