I've got a dataset of 1.5 million and am looking to train a classifier (probably RF to start) -- I have 7 classes to predict and 20K text features. Like most distributions of text features, only 20% of them account for 80% of occurrences in the sample. I am going to manually label 10K of the sample to predict the classes for the rest 1.5 million.
My question is, how would I choose the subsample based on the features and distribution. Should I just choose a random sample (ie try to match the distribution)? Or should I try to find the 10K that maximizes the number of features represented in the sample? Whats the benefit and drawback of each?
I have only one shot to label these 10K so I want to make sure I choose the right sample!