7
$\begingroup$

Is there a difference between doing preprocessing for a dataset in sklearn before and after splitting data into train_test_split?

In other words, are both of these approaches equivalent?

from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split

#standardizing after splitting
X_train, X_test, y_train, y_test = train_test_split(data, target)
sc = StandardScaler().fit(X_train)
X_train_std = sc.transform(X_train)
X_test_std = sc.transform(X_test)

#standardizing before splitting
data_std = StandardScaler().fit_transform(data)
X_train, X_test, y_train, y_test = train_test_split(data_std, target)
$\endgroup$
2
$\begingroup$

No, Both approaches are not equivalent.

StandardScaler() standardize features by removing the mean and scaling to unit variance

If you fit the scaler after splitting: Suppose, if there are any outliers in the test set(after Splitting), the Scaler would not consider those in computing mean and Variance.

If you fit the scaler on whole dataset and then split, Scaler would consider all values while computing mean and Variance.

Since, the mean and variance are different in both cases, the fits and transform functions would perform differently.

$\endgroup$
  • $\begingroup$ Thanks for the answer. So if I understand well, the first approach doesn't make sense so it must not be used, am I right? $\endgroup$ – W.R. Mar 31 '17 at 9:24
  • $\begingroup$ Yeah, I would suggest you to fit the scaler with whole data, because you are just "modifying the way the the data is presented" or encoded. There is no effect on test data because, you are "not modifying the data." $\endgroup$ – phanny Mar 31 '17 at 10:09
  • 3
    $\begingroup$ @phanny but by using test data to scale the training data, you are "peeking" at the test data and using some of the information about what the test data is like during training. Yes, the effect of that is probably going to be insignificant, but is it not prudent to leave the test data alone, "hide it in a safety deposit box", until you are actually testing your final model? $\endgroup$ – rinspy Mar 31 '17 at 18:17
  • 3
    $\begingroup$ The test set is data that you should ideally not even have available while you are selecting, training and optimising your model. I think your comments above are somewhat misleading. $\endgroup$ – rinspy Mar 31 '17 at 21:01
  • 3
    $\begingroup$ @phanny Cross-validation is done on the training set. The test set should not be used until the final stage of creating the model, and should only used to estimate the model's out-of-sample performance. In any case, in cross-validation, standardization of features should be done on training and validation sets in each fold separately. Consider, for example, what happens if your data is non-stationary and your validation set has a different distribution of features than your training set. In this case, by standardizing in advance, you will get inflated performance in CV. $\endgroup$ – rinspy Nov 3 '17 at 12:01
7
$\begingroup$

To add to phanny's answer - you should do the preprocessing on your training set separately, otherwise information from the test set will "leak" into your training data.

For preprocessing the test set, I do not see why you shouldn't preprocess it together with your training data.

$\endgroup$
  • $\begingroup$ Thanks rinspy, please could you elaborate more? As I understood from phanny's answer, the preprocessing should be done on the whole training data $\endgroup$ – W.R. Mar 31 '17 at 9:34
  • $\begingroup$ I added a comment on the other answer; I am curious myself. I think for cross-validation it is fine to do as @phanny suggested, but I do think you should not look at or use your final test data set in any way until it is time to actually test your final, tuned, hyper-parameter-optimized model. $\endgroup$ – rinspy Mar 31 '17 at 18:18
2
$\begingroup$

The test set should ideally not be preprocessed with the training data. This will ensure no 'peeking ahead'. Train data should be preprocessed separately and once the model is created we can apply the same preprocessing parameters used for the train set, onto the test set as though the test set didn't exist before.

$\endgroup$
0
$\begingroup$

You can also use the method below will preprocess your data separately but similar parameter used for training data set.

norm = preprocessing.Normalizer().fit(xtrain)

then

x_train_norm = normalizer.transform(xtrain) 
x_test_norm = normalizer.transform(Xtest)
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.