best practice: preprocessing holdout set at same time as train set, or no?

Question

I am a very new ML programmer, and I have come across a dilemma regarding best practices. Things will "work" either way for me, but I want to know what the best practice is.

I am performing text classification, and I need to perform certain preprocessing steps. For example I need to run my text through CountVectorizer (I am using scikit-learn) or something similar, to convert it to a vector representation.

The way I had learned from a book, you split the data into train/test groups first, and then afterward you can do transformations:

my_text_data, my_targets = get_project_data()
text_train, text_test, y_train, y_test = train_test_split(
    my_text_data, my_targets, random_state=0)

# Now we will vectorize our training data:
vect = CountVectorizer()
vect.fit(text_train)
X_train = vect.transform(text_train)

# ...
# Later on when we are testing, we will need to vectorize our test data
X_test = vect.transform(text_test)

I realized that technically, I can just do this:

my_text_data, my_targets = get_project_data()

# Just transform all of our data (train and test) at once. 
vect = CountVectorizer()
vect.fit(my_text_data)
X_all = vect.transform(my_text_data)

X_train, X_test, y_train, y_test = train_test_split(
    X_all, my_targets, random_state=0)

One upside is that I wouldn't be duplicating these transformation calls. And I'd be guaranteed not to make an error by transforming the test data in a different way than otherwise.

However I think in some cases this could lead to subtly different results, because if I used TfidfVectorizer instead of CountVectorizer, then the "inverse document frequency" calculation during fitting would be taking into account document frequencies among the holdout set in addition to the training set. So maybe that's a good example of where this would violate the separation between training/holdout sets, and shouldn't be done.

Curious for your thoughts... thanks!

Answer 1

In general, the holdout set should indeed be held out during training phase. Here you exhibit a nice example to a case where including the holdout (test) set in the preprocessing phase would influence the preprocessing and thus affect the training results!

The way to go is doing preprocessing on the training set, training your model, then preprocess the holdout set (in the exact same way), and predict using the trained model.

Let me include a simple example:

My preprocessing phase in the model is removing the mean value and dividing by the STD. When I preprocess the test set I would subtract the mean value of the training set and divide by the training STD. The exact same process is done on the training and on the test set, the mean and STD aren't calculated again at test time.

Hope this helps!