2
$\begingroup$

In my actual data class A has 90%, class B has 9% and class C has 1% (numbers are made up for sake of simplicity). Now I want to prepare a training set for my classifier (I plan to use Vowpal Wabbit). How should I distribute entries according to classes? Should I maintain the same ratio as in real data or I should distribute entries evenly?

$\endgroup$
2
$\begingroup$

Answering your second question first:

Should I maintain the same ratio as in real data or I should distribute entries evenly?

In my opnion, it makes sense to maintain the same proportion of classes as in the complete set of data. Typically, there is a reason why this distribution is skewed (e.g., it may be naturally observed in the event being studied) and hence learning in the presence of this skew would help to generalize better to the unseen samples. This would be my suggestion in the general case. Subjectively, things could be different. And, of course, you will then have to be careful about how you evaluate the performance of your classifier. There is literature on how to handle such skewed cases.

Now, to answer your first question:

How should I distribute entries according to classes?

You could use stratified folds to prepare your training sets. This will let you maintain the proportion of the classes in your training folds as in the complete data.

I am not sure if you work with Python, but to get you started, scikit-learn, a popular Python package for machine learning, has this provision for you in sklearn.cross_validation.StratifiedKFold.

$\endgroup$
  • $\begingroup$ The problem I see here that some classes have rather rare occurrence and I will need very big training set to give them good representation. $\endgroup$ – Andrey Sep 24 '14 at 21:59
  • $\begingroup$ @Andrey, right. In cases where this may be difficult (to get a big training set) or impossible, one will have to think of feasible ways/work-arounds to deal with it. $\endgroup$ – SPN Sep 24 '14 at 22:07
1
$\begingroup$

There are several ways to work around class-imbalance. As SPN pointed out , its probably best to maintain the class distribution in your training data as per your real data.

Also want to point another way to doing this: working with a class distribution of 90-9-1 could be a lot trickier than working with 90-10, ie. combining classes B and C and have your classifier work on the "A or not A" classification problem. Then, build a second classifier to classify between B and C, if required. The good thing about working with binary classifiers (2-class) is that there are alternative measures of accuracy (eg. F-measure, adjusted geometric mean (http://www.ncbi.nlm.nih.gov/pubmed/22809416)).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.