I'm a data science newbie and a bit confused with the following:
I usually do the preprocessing on all predictors of a dataset, meaning
I create X
by concatenating X_train
and X_test
.
(Imagine a competition where you download test and training data separately.)
After the preprocessing I use scikit's train_test_split
to split the data into train and test data.
I was wondering if doing the preprocessing on X
altogether can lead to raintrain-test contamination or target leakage.
I saw that you shouldn't do
X_valid = imputer.fit_transform(X_valid)
for example. And that
X_train = imputer.fit_transform(X_train)
X_valid = imputer.transform(X_valid)
is a better option.