In order to avoid data leakage, when preprocessing data, we need to
- Step1: split the dataset into train, validation, and test set.
- Step2: fit a transformer on train set and transform train, validation, and test set with this transformer.
For example, when doing standardization, when $\mu$'s and $\sigma$'s for each feature are estimated on train set, the train, validation, and test set are transformed with these $\mu$'s and $\sigma$'s.
As much as this makes sense, especially when test data comes in stream, I am not sure why this is the practice when people are working with existing dataset (i.e. when people have all the data they need) and given that
- Dataset is randomly split and outliers could occur in any subset (for example, using
- Outliers could be removed before processing even happens.