16
$\begingroup$

Is there a difference between doing preprocessing for a dataset in sklearn before and after splitting data into train_test_split?

In other words, are both of these approaches equivalent?

from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split

#standardizing after splitting
X_train, X_test, y_train, y_test = train_test_split(data, target)
sc = StandardScaler().fit(X_train)
X_train_std = sc.transform(X_train)
X_test_std = sc.transform(X_test)

#standardizing before splitting
data_std = StandardScaler().fit_transform(data)
X_train, X_test, y_train, y_test = train_test_split(data_std, target)
Improve this question
$\endgroup$

4 Answers 4

Reset to default
5
$\begingroup$

No, Both approaches are not equivalent.

StandardScaler() standardize features by removing the mean and scaling to unit variance

If you fit the scaler after splitting: Suppose, if there are any outliers in the test set(after Splitting), the Scaler would not consider those in computing mean and Variance.

If you fit the scaler on whole dataset and then split, Scaler would consider all values while computing mean and Variance.

Since, the mean and variance are different in both cases, the fits and transform functions would perform differently.

Improve this answer
$\endgroup$
8
  • $\begingroup$ Thanks for the answer. So if I understand well, the first approach doesn't make sense so it must not be used, am I right? $\endgroup$
    – W.R.
    Commented Mar 31, 2017 at 9:24
  • $\begingroup$ Yeah, I would suggest you to fit the scaler with whole data, because you are just "modifying the way the the data is presented" or encoded. There is no effect on test data because, you are "not modifying the data." $\endgroup$
    – phanny
    Commented Mar 31, 2017 at 10:09
  • 9
    $\begingroup$ @phanny but by using test data to scale the training data, you are "peeking" at the test data and using some of the information about what the test data is like during training. Yes, the effect of that is probably going to be insignificant, but is it not prudent to leave the test data alone, "hide it in a safety deposit box", until you are actually testing your final model? $\endgroup$
    – rinspy
    Commented Mar 31, 2017 at 18:17
  • 7
    $\begingroup$ The test set is data that you should ideally not even have available while you are selecting, training and optimising your model. I think your comments above are somewhat misleading. $\endgroup$
    – rinspy
    Commented Mar 31, 2017 at 21:01
  • 8
    $\begingroup$ @phanny Cross-validation is done on the training set. The test set should not be used until the final stage of creating the model, and should only used to estimate the model's out-of-sample performance. In any case, in cross-validation, standardization of features should be done on training and validation sets in each fold separately. Consider, for example, what happens if your data is non-stationary and your validation set has a different distribution of features than your training set. In this case, by standardizing in advance, you will get inflated performance in CV. $\endgroup$
    – rinspy
    Commented Nov 3, 2017 at 12:01
11
$\begingroup$

To add to phanny's answer - you should do the preprocessing on your training set separately, otherwise information from the test set will "leak" into your training data.

For preprocessing the test set, I do not see why you shouldn't preprocess it together with your training data.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thanks rinspy, please could you elaborate more? As I understood from phanny's answer, the preprocessing should be done on the whole training data $\endgroup$
    – W.R.
    Commented Mar 31, 2017 at 9:34
  • 1
    $\begingroup$ I added a comment on the other answer; I am curious myself. I think for cross-validation it is fine to do as @phanny suggested, but I do think you should not look at or use your final test data set in any way until it is time to actually test your final, tuned, hyper-parameter-optimized model. $\endgroup$
    – rinspy
    Commented Mar 31, 2017 at 18:18
  • 2
    $\begingroup$ No, even during cross validation, you should do the preprocessing inside the cross validation loop - the same principle of not peeking ahead applies. The only exception are preprocessing steps that work on each case independent of all other cases (i.e. row-wise transformations) - they can safely be taken outside / before the CV loop. $\endgroup$
    – cbeleites
    Commented Mar 12, 2020 at 19:44
4
$\begingroup$

The test set should ideally not be preprocessed with the training data. This will ensure no 'peeking ahead'. Train data should be preprocessed separately and once the model is created we can apply the same preprocessing parameters used for the train set, onto the test set as though the test set didn't exist before.

Improve this answer
$\endgroup$
2
$\begingroup$

You can also use the method below will preprocess your data separately but similar parameter used for training data set. 

norm = preprocessing.Normalizer().fit(xtrain)

then

x_train_norm = norm.transform(xtrain) 
x_test_norm = norm.transform(Xtest)
Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.