Cross-validation transformer fit to test set?

Question

When applying transformers in a cross-validation routine, it is often advised to fit the transformer to the data in your train set, and transform both the train and test set using the obtained transformer parameters. As an example, suppose we are using a standard scaler as a transformer, the cross-validation routine might look like this:

from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import KFold

scaler = StandardScaler()
cv = KFold(n_splits=5)
folds = cv.split(X=X)

for train_idx, test_idx in folds:
    X_train = X[train_idx,:]
    X_test = X[test_idx,:]
    y_train = y[train_idx]
    y_test = y[test_idx]

    scaler.fit(X_train)
    X_train = scaler.transform(X_train)
    X_test = scaler.transform(X_test)

    # Train & score model

This is also the behavior that Scikitlearn's pipelines implement.

What I'm interested in is the following: Why do we transform the test set based on the parameters of the train set, instead of fitting a separate transformer for the test set? If we would fit a separate standard scaler based on the test set and use the obtained parameters to transform the test set, then, as far as I see, the train and test set remain independent. As an example, we could do the following in every fold:

X_train = StandardScaler().fit_transform(X_train)
X_test = StandardScaler().fit_transform(X_test)

In case of a standard scaler, it would probably hardly matter for the performance of the resulting model. But I can imagine there to be transformers in which it matters. Is there any objection to using the second method described here?

Answer 1

This answer is the same as what @Hooman said, but explained a little differently. For this example I'm going to pretend you have users visiting your site, and you want to predict if they will click on an ad or not.

The idea of a train-test split is that you want to mimic real life. In real life you will have some historical data about users who came to your site, and you will know if they clicked on an ad or not, this is your training set. You now have to build a model to predict if future users will click on an ad, but you don't have this data yet, this is your test set. You can't use the mean and standard deviation from the test set yet because you wont get that data until tomorrow, and you need to build a model tonight!

Answer 2

Let's take a very basic example:

X_train = [-4, -3, -2, -1, 0, 1, 2, 3, 4]

Y_train = [-8, -6, -4, -2, 0, 2, 4, 6, 8]

X_test = [3.5, 4.5]

Y_test = [7, 9]

As you can see, the model $y=2x$ works perfectly for this data. To simplify, let's say we want to center the data, but we don't want to normalize the standard deviation to 1. We can use a standard scaler on X_train. Actually, it is already centered, so it will not change anything. If we use another scaler for the test set, we are going to obtain: Xtest_modified = [-0.5, 0.5]. And after training we are going to predict: Y_test = [-1 ,1] which is obviously wrong.

If you had used the same scaler (which does not do anything in this case), you would have obtained Xtest_modified=[3.5, 4.5], which gives correct estimates.

Although in this case, the error is huge, it may be that with large dataset the error is almost invisible. Indeed, if you have a large i.i.d. dataset, the mean and variance of your training and test sets will be very similar.

Answer 3

as far as I see, the train and test set remain independent

You're right that information from the test set wouldn't be leaking into the training set, but it could be leaking into the model at test time.

In case of a standard scaler, it would probably hardly matter for the performance of the resulting model.

Transformers that are fit to a dataset contain information about that dataset. So if you fit a transformer (such as StandardScaler) to the test set, then information from the test data (mean and standard deviation in this case) has leaked into that transformer. This information leakage may cause your model to overfit the test set.