I've come across the concept of data leakage in which optimistically biased generalisation errors occur due to test data in some sense 'seeing' the training data. For instance, normalisation on an entire dataset before train/test splitting occurs means the test set was normalised using parameters influenced by the training set.
I have been convinced, by this blog, to avoid it by performing train/test splits first, transforming the train data and then transforming the test data using parameters from the train set.
What is the advantage of using train parameters to transform the test set? Is it simply that the train set was larger and so a better estimate of the population parameters? What would be the consequences of using the test parameters to transform the test set - it avoids leakage, but perhaps has worse estimates for parameters and so performs more poorly?
I have an application (vibrational spectroscopy) in which some transformations (e.g. normalisation) occur at the per sample level (e.g. using the mean/std of the sample rather than the global). In this case I believe data leakage is not a concern, correct?
Does anyone know any published papers that explicitly look at data leakage and the effect it has on generalisation error? This is the best one I found, but couldn't find anything more.