We are doing a lot of experiments in my research group with text data, and what usually happens is that a corpus will be transformed into instances with features as bag of word or n-gram features. We then perform machine learning and evaluate the models using cross validation. It seems as though you shouldn't do this, since you don't really know what features will be in each fold of the CV until you actually run it. Therefore, you should be creating the word vector on training folds, otherwise you are including test data when making your features. Here is a toy example:
If we have two sentences (tweets, or any string) that make up our dataset:
- I love chicken and rice
- Taco Bell chicken burrito is good
Cross validation would then split these into two different datasets, sentence 1 for training, and sentence 2 for testing, for example. If we build the word vector before the cross validation happens, the features would be:
love, chicken, rice, taco, bell, burrito, good
But, in reality, for our cross validation, the features should only be:
love, chicken, rice
Because sentence 1 was the training set for that run of CV. Sentence 2 is part of the test set, so including the words from sentence 2 would be “cheating”.
Any thoughts? I've looked around online and in literature and haven't found much discussion on this.