0
$\begingroup$

I've got a dataset of 1.5 million and am looking to train a classifier (probably RF to start) -- I have 7 classes to predict and 20K text features. Like most distributions of text features, only 20% of them account for 80% of occurrences in the sample. I am going to manually label 10K of the sample to predict the classes for the rest 1.5 million.

My question is, how would I choose the subsample based on the features and distribution. Should I just choose a random sample (ie try to match the distribution)? Or should I try to find the 10K that maximizes the number of features represented in the sample? Whats the benefit and drawback of each?

I have only one shot to label these 10K so I want to make sure I choose the right sample!

$\endgroup$
0
$\begingroup$

Whenever you pick a highly specialized sample, you are running the risk of being utterly wrong with it. Gary King here argued that humans are awful at coming up with keywords. So if you miss on these in the beginning, your classifier may not work that well out of sample. (He also had another paper that's more closely related to your task of working with subsamples and recovering unbiased estimates of population proportions after manual coding.)

$\endgroup$
  • $\begingroup$ Thanks. so basically a random sample is safest, but anything else might run the risk of being devastatingly wrong? If that's so, I'll probably go with random as it'll be a considerable amount of effort to label the 10K, and don't want to mess that up.. $\endgroup$ – dislogic Apr 18 '17 at 17:43
  • $\begingroup$ Well, in many applications, well informed non-i.i.d. samples greatly improve analysis. In your case, it seems like a strategy you were thinking of is to make your sample more diverse than the population, i.e., undersample the most common features and oversample the rare ones. If you think you need only 20 samples to train your algorithm to classify "textbook" as having to do with "education" rather than 100 that would come out of proportional representation, that could make sense, too. $\endgroup$ – StasK Apr 19 '17 at 20:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.