0
$\begingroup$

Question

In order to avoid data leakage, when preprocessing data, we need to

  • Step1: split the dataset into train, validation, and test set.
  • Step2: fit a transformer on train set and transform train, validation, and test set with this transformer.

For example, when doing standardization, when $\mu$'s and $\sigma$'s for each feature are estimated on train set, the train, validation, and test set are transformed with these $\mu$'s and $\sigma$'s.

As much as this makes sense, especially when test data comes in stream, I am not sure why this is the practice when people are working with existing dataset (i.e. when people have all the data they need) and given that

  • Dataset is randomly split and outliers could occur in any subset (for example, using train_test_split() in sklearn).
  • Outliers could be removed before processing even happens.
$\endgroup$
1
$\begingroup$

This is not about data streams or incomplete / yet-unavailable new data. Your data needs to be on the same scale in all data sets - even if the scale is off, as long as it's the same - for the model to work as planned, which is why you only use your training parameters to transform all the other data sets, since you used these parameters to transform your train set, upon which you built your model. So any new data needs to be in the form that the model saw during training. Otherwise you may get a $\mu$ of 2 in the train set and a $\mu$ of 5 in the test set, which would completely change the transformation of the test data and hence the model output.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.