We are doing a lot of experiments in my research group with text data, and what usually happens is that a corpus will be transformed into instances with features as bag of word or n-gram features. We then perform machine learning and evaluate the models using cross validation. It seems as though you shouldn't do this, since you don't really know what features will be in each fold of the CV until you actually run it. Therefore, you should be creating the word vector on training folds, otherwise you are including test data when making your features. Here is a toy example:

If we have two sentences (tweets, or any string) that make up our dataset:

  1. I love chicken and rice
  2. Taco Bell chicken burrito is good

Cross validation would then split these into two different datasets, sentence 1 for training, and sentence 2 for testing, for example. If we build the word vector before the cross validation happens, the features would be:

love, chicken, rice, taco, bell, burrito, good

But, in reality, for our cross validation, the features should only be:

love, chicken, rice

Because sentence 1 was the training set for that run of CV. Sentence 2 is part of the test set, so including the words from sentence 2 would be “cheating”.

Any thoughts? I've looked around online and in literature and haven't found much discussion on this.


The purpose of cross validation is to simulate how your model would perform on unseen data, including data that contains words or n-grams that were not contained in any of your training data. This motivates the following steps:

  • Assign data to CV folds
  • Create features based on training fold and train model
  • Extract identical features from the testing dataset and apply the model
  • Repeat for all k folds. The average error is an approximation of the generalized error rate.

Incorporating outside data is not needed. Consider these possibilities:

  • The word is in the training dataset: These words will already show up in your model without using outside data.
  • The word is not in the training or the testing dataset: These words unnecessarily add more features to the dataset. If you remove any because they don't occur in the training or testing dataset, then you are using your testing dataset for feature selection which is not good practice and can bias your model.
  • The word is only in the testing dataset: Even if this feature is in your training dataset because you added it from an outside corpus, all the values of feature will be set to zero for every observation in your training data. This means that there is no information that any possible model could extract from the feature being included in the training dataset.

For all these possibilities, collecting an outside corpus adds additional work, without any benefit above standard cross validation.

I would recommend setting the number of cv folds to be as high as your computing resources allow if you really want to minimize the impact of excluded features. Consider if you use 20 fold cv. Then for a feature to not appear in one of your folds, at worst it would occur in only 5% of your observations. But, the feature would be guaranteed to occur in the other 19 folds. In practice, the issue of leaving a few rare words out of a single cv does very little to the predictive accuracy of the model.

  • $\begingroup$ we have found that it take more computing resources to re-compute the word vector in each fold, so preprocessing the data once would speed things up. So you're saying that with a high number of folds, it should be ok to preprocess? $\endgroup$ – rikturr Jun 15 '15 at 18:30
  • $\begingroup$ It really depends what you are doing in the preprocessing step. Most of the time I will optimize hyperparameters in the word vectorization step. If you intend to do this as well, you must bare the extra computing cost. If you intend to only do a single transformation of your dataset and this transformation does not incorporate any of the y values, then you should be ok preprocessing before cross validation. $\endgroup$ – Jason Sanchez Jun 15 '15 at 22:55

you bring up a good point, but in fact both are incorrect. The text corpus that you use should be large enough to include as many words as possible. It should NOT just include your data. Training on a bunch of wikipedia articles and random tweets is a good way to do this. This will give you a dictionary, where each word corresponds to a vector.

Now in order to do anything you have to use that dictonary as a reference for your text. Now you can do your machine learning (I'm assuming you are keeping that vague on purpose). If you aren't doing it already, try using a convnet on the vectors...

You want to use a large dataset for learning the dictonary (of vectors). You want to have a feature space that is large enough to separate into whatever you're trying to classify. However, if you only vectorize the training data, you will miss important words.

Your current procedure sort of defeats the purpose of cross validation, since cross validation is suppose to simulate data that you haven't seen.... thats why you want a very large training corpus that is independent of your training and test data....


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.