I am working on a prediction problem where I was given a 6,000 record dataset with the value of the dependent variable included ("train"), and a 2,000 record dataset with the same independent variables but without the dependent variable ("test"). A couple of the independent variables are (longish) text and so I've constructed a Document Term Matrix (DTM) using bag of words, keeping only words that appear in >2% of records (reducing my words-features from ~700 to ~50). I originally created my DTM based on the 6,000 train records I have a dependent variable for, to apply supervised learning methods.
However, since I know I will be scored on predictions for the 2,000 record test set, I started thinking I am making a mistake by not including words from those 2,000 records in my DTM. So I'm considering recalculating my DTM using all 8,000 records, and then use my 6,000 record training set (or rather 70% split of my 6,000 record training set) to determine which subset of words to keep as features. So call this plan A: maybe I should be pulling in more records to construct my DTM.
But then I've been reading a lot about the importance of including feature selection in cross-validation, especially when you whittle down many potential variables to a much smaller number to include in the model. I realized that's what I do with my DTM, when I filter to keep only the words which appear in at least 2% of records. So now I'm wondering if, to do 5-fold Cross Validation correctly - should I be constructing 5 different DTMs, keeping a different set of top features for each fold/DTM? So call this plan B: maybe I should be creating DTMs based on way fewer records than I did originally (e.g. each DTM based on 1/5 of 6,000 records for 5-fold CV).
Based on a lot of thought, here's what I think I think I should do:
- Create a Document Term Matrix based on all 8,000 records I have, but do not whittle down words/features at this step. This will leave me with ~800 words in my DTM.
- Apply 5-fold Cross Validation where, for each fold, I select features by keeping only words that appear in >2% of records in that fold. So I will have 5 DTMs with different feature sets, though they will all have been whittled down from the same set of features.
I think this best provides me the balance of: a) including information in the 2,000 dataset I will be tested on, while b) including the key feature-selection step in my Cross Validation. But I'm (very) new to this and would be very interested in anyone else's opinion here.