# best practice: preprocessing holdout set at same time as train set, or no?

I am a very new ML programmer, and I have come across a dilemma regarding best practices. Things will "work" either way for me, but I want to know what the best practice is.

I am performing text classification, and I need to perform certain preprocessing steps. For example I need to run my text through CountVectorizer (I am using scikit-learn) or something similar, to convert it to a vector representation.

The way I had learned from a book, you split the data into train/test groups first, and then afterward you can do transformations:

my_text_data, my_targets = get_project_data()
text_train, text_test, y_train, y_test = train_test_split(
my_text_data, my_targets, random_state=0)

# Now we will vectorize our training data:
vect = CountVectorizer()
vect.fit(text_train)
X_train = vect.transform(text_train)

# ...
# Later on when we are testing, we will need to vectorize our test data
X_test = vect.transform(text_test)


I realized that technically, I can just do this:

my_text_data, my_targets = get_project_data()

# Just transform all of our data (train and test) at once.
vect = CountVectorizer()
vect.fit(my_text_data)
X_all = vect.transform(my_text_data)

X_train, X_test, y_train, y_test = train_test_split(
X_all, my_targets, random_state=0)


One upside is that I wouldn't be duplicating these transformation calls. And I'd be guaranteed not to make an error by transforming the test data in a different way than otherwise.

However I think in some cases this could lead to subtly different results, because if I used TfidfVectorizer instead of CountVectorizer, then the "inverse document frequency" calculation during fitting would be taking into account document frequencies among the holdout set in addition to the training set. So maybe that's a good example of where this would violate the separation between training/holdout sets, and shouldn't be done.

Curious for your thoughts... thanks!

In general, the holdout set should indeed be held out during training phase. Here you exhibit a nice example to a case where including the holdout (test) set in the preprocessing phase would influence the preprocessing and thus affect the training results!

The way to go is doing preprocessing on the training set, training your model, then preprocess the holdout set (in the exact same way), and predict using the trained model.

Let me include a simple example:

My preprocessing phase in the model is removing the mean value and dividing by the STD. When I preprocess the test set I would subtract the mean value of the training set and divide by the training STD. The exact same process is done on the training and on the test set, the mean and STD aren't calculated again at test time.

Hope this helps!

• Here is one example where I wasn't able to avoid preprocessing. Not sure if I did the right thing. I am building a multilabel classifier. To represent my target values, I need an indicator array (i.e. each sample has an array with 0's for the labels it doesn't have, and 1's for the labels it does). To generate this I use MultiLabelBinarizer().fit_transform(raw_targets). I tried calling this separately on y_train and y_test, but since the number of columns don't match, the scoring algorithm on my meta-estimator OneVsRestClassifier complains and won't proceed. – Stephen Aug 15 '17 at 1:51
• In other words, you need a consistent number of columns in your array of target values. To solve it this time, I built the array before splitting between training and test. Perhaps the alternative here would be to save the MultiLabelBinarizer object, and essentially get it to ignore any target values in the test set which weren't seen in the training set. I haven't looked into how to do that yet. – Stephen Aug 15 '17 at 1:53
• I don't really get what you mean. The training and the test set must include all kinds of labels and thus the number of columns should be exactly the same. If this is not the case then you should reorgenize the data so both sets would include examples from all classes. Moreover, it is beneficial if all classes have similar representation for each set i.e.: if you have [100, 90, 120] examples in class [A,B,C], respectively it is much better than [10, 200, 159] (unbalanced classes). – AR_ Aug 15 '17 at 5:08