I am a very new ML programmer, and I have come across a dilemma regarding best practices. Things will "work" either way for me, but I want to know what the best practice is.
I am performing text classification, and I need to perform certain preprocessing steps. For example I need to run my text through CountVectorizer (I am using scikit-learn) or something similar, to convert it to a vector representation.
The way I had learned from a book, you split the data into train/test groups first, and then afterward you can do transformations:
my_text_data, my_targets = get_project_data() text_train, text_test, y_train, y_test = train_test_split( my_text_data, my_targets, random_state=0) # Now we will vectorize our training data: vect = CountVectorizer() vect.fit(text_train) X_train = vect.transform(text_train) # ... # Later on when we are testing, we will need to vectorize our test data X_test = vect.transform(text_test)
I realized that technically, I can just do this:
my_text_data, my_targets = get_project_data() # Just transform all of our data (train and test) at once. vect = CountVectorizer() vect.fit(my_text_data) X_all = vect.transform(my_text_data) X_train, X_test, y_train, y_test = train_test_split( X_all, my_targets, random_state=0)
One upside is that I wouldn't be duplicating these transformation calls. And I'd be guaranteed not to make an error by transforming the test data in a different way than otherwise.
However I think in some cases this could lead to subtly different results, because if I used TfidfVectorizer instead of CountVectorizer, then the "inverse document frequency" calculation during fitting would be taking into account document frequencies among the holdout set in addition to the training set. So maybe that's a good example of where this would violate the separation between training/holdout sets, and shouldn't be done.
Curious for your thoughts... thanks!