Should you create a word vector before cross validation?

Question

We are doing a lot of experiments in my research group with text data, and what usually happens is that a corpus will be transformed into instances with features as bag of word or n-gram features. We then perform machine learning and evaluate the models using cross validation. It seems as though you shouldn't do this, since you don't really know what features will be in each fold of the CV until you actually run it. Therefore, you should be creating the word vector on training folds, otherwise you are including test data when making your features. Here is a toy example:

If we have two sentences (tweets, or any string) that make up our dataset:

1. I love chicken and rice
2. Taco Bell chicken burrito is good

Cross validation would then split these into two different datasets, sentence 1 for training, and sentence 2 for testing, for example. If we build the word vector before the cross validation happens, the features would be:

love, chicken, rice, taco, bell, burrito, good

But, in reality, for our cross validation, the features should only be:

love, chicken, rice

Because sentence 1 was the training set for that run of CV. Sentence 2 is part of the test set, so including the words from sentence 2 would be “cheating”.

Any thoughts? I've looked around online and in literature and haven't found much discussion on this.

Answer 1

The purpose of cross validation is to simulate how your model would perform on unseen data, including data that contains words or n-grams that were not contained in any of your training data. This motivates the following steps:

• Assign data to CV folds
• Create features based on training fold and train model
• Extract identical features from the testing dataset and apply the model
• Repeat for all k folds. The average error is an approximation of the generalized error rate.

Incorporating outside data is not needed. Consider these possibilities:

• The word is in the training dataset: These words will already show up in your model without using outside data.
• The word is not in the training or the testing dataset: These words unnecessarily add more features to the dataset. If you remove any because they don't occur in the training or testing dataset, then you are using your testing dataset for feature selection which is not good practice and can bias your model.
• The word is only in the testing dataset: Even if this feature is in your training dataset because you added it from an outside corpus, all the values of feature will be set to zero for every observation in your training data. This means that there is no information that any possible model could extract from the feature being included in the training dataset.

For all these possibilities, collecting an outside corpus adds additional work, without any benefit above standard cross validation.

I would recommend setting the number of cv folds to be as high as your computing resources allow if you really want to minimize the impact of excluded features. Consider if you use 20 fold cv. Then for a feature to not appear in one of your folds, at worst it would occur in only 5% of your observations. But, the feature would be guaranteed to occur in the other 19 folds. In practice, the issue of leaving a few rare words out of a single cv does very little to the predictive accuracy of the model.

Answer 2

you bring up a good point, but in fact both are incorrect. The text corpus that you use should be large enough to include as many words as possible. It should NOT just include your data. Training on a bunch of wikipedia articles and random tweets is a good way to do this. This will give you a dictionary, where each word corresponds to a vector.

Now in order to do anything you have to use that dictonary as a reference for your text. Now you can do your machine learning (I'm assuming you are keeping that vague on purpose). If you aren't doing it already, try using a convnet on the vectors...

You want to use a large dataset for learning the dictonary (of vectors). You want to have a feature space that is large enough to separate into whatever you're trying to classify. However, if you only vectorize the training data, you will miss important words.

Your current procedure sort of defeats the purpose of cross validation, since cross validation is suppose to simulate data that you haven't seen.... thats why you want a very large training corpus that is independent of your training and test data....