Skip to main content
2 of 3
edited title
LeLuc
  • 671
  • 5
  • 11

Avoiding data leakage in preprocessing

I'm a data science newbie and a bit confused with the following:
I usually do the preprocessing on all predictors of a dataset, meaning I create X by concatenating X_train and X_test. (Imagine a competition where you download test and training data separately.)

After the preprocessing I use scikit's train_test_split to split the data into train and test data.

I was wondering if doing the preprocessing on X altogether can lead to rain-test contamination or target leakage.

I saw that you shouldn't do

X_valid = imputer.fit_transform(X_valid)

for example. And that

X_train = imputer.fit_transform(X_train)
X_valid = imputer.transform(X_valid)

is a better option.

LeLuc
  • 671
  • 5
  • 11