Data Leakage Concerns

Question

I've come across the concept of data leakage in which optimistically biased generalisation errors occur due to test data in some sense 'seeing' the training data. For instance, normalisation on an entire dataset before train/test splitting occurs means the test set was normalised using parameters influenced by the training set.

I have been convinced, by this blog, to avoid it by performing train/test splits first, transforming the train data and then transforming the test data using parameters from the train set.

Some questions:

What is the advantage of using train parameters to transform the test set? Is it simply that the train set was larger and so a better estimate of the population parameters? What would be the consequences of using the test parameters to transform the test set - it avoids leakage, but perhaps has worse estimates for parameters and so performs more poorly?

I have an application (vibrational spectroscopy) in which some transformations (e.g. normalisation) occur at the per sample level (e.g. using the mean/std of the sample rather than the global). In this case I believe data leakage is not a concern, correct?

Does anyone know any published papers that explicitly look at data leakage and the effect it has on generalisation error? This is the best one I found, but couldn't find anything more.

Answer 1

... test data in some sense 'seeing' the training data ...

The situation in data leakage is actually training data seeing test data, such that the test data leaks into your modelling process and affects the performance calculations. When you use training set statistics (e.g. mean/std) while standardising the test set, it's not called data leakage. Training set is always there since the model is dependent on that.

What is the advantage of using train parameters to transform the test set?

Normalisation/standardisation is also a part of the modelling process, and it should be completed before testing. Imagine your model is deployed and making predictions online (e.g. on user devices), you won't have access to aggregate test statistics because the test set is distributed across many devices online, so you use training statistics. The purpose of the test set is to emulate this behaviour so that you can have an idea of how your model will perform when it's deployed. Using test set statistics in your evaluation wouldn't be fair since you won't be able to do that in runtime.

Note that even if you have the test set (in runtime), you should be using the training set stats because otherwise would mean interfering with the model. Above mentioned scenario is just a good example.

In your particular case where the standardisation takes place per sample, it is domain specific, and not related to the data leakage mentioned above.