In machine learning, Is it better to have class ratios balanced or representative of the population?

Question

In the context of machine learning, let's say you have a problem in which classes in the real population are not balanced - eg Class A occurs 80% of the time and Class B occurs 20% of the time.

In such a case, is it generally better to have a given ML algorithm rely on data with the same 80/20 class ratio, or data with a balanced (50/50) ratio? a) with regards to training data b) with regards to test data

A followup question: In case the answer for (a) or (b) happens to be going with the balanced 50/50 ratio, then does this preference generally still persist even in the practical context where the data one has access to happens to be of the 80/20 ratio? In other words, would the benefit of using a balanced ratio to train and/or test outweigh the cost of enforcing that ratio (e.g. by discarding instances from the majority class or generating new synthetic samples of the minority class)?

Answer 1

Check this paper for a good review of learning with inbalanced datasets.

One way of dealing with the problem is to do artificial subsampling or upsampling in the training set to balance the datasets.

I think it is usually better to have a balanced training set, since otherwise the decision boundary is gonna give too much space to the bigger class and you are going to misclassify too much the small class. This is usually bad. (think of cancer detection where the smaller class is the most costly, namely having a tumor).

If you don't want to use sampling methods, than you can use cost based methods, where you weight the importance of every sample so that the loss function has more contribution from the samples of the most important class. In cancer detection, you would weight more the cost coming from training samples of hte positive class (having a tumor).

Finally, remember that if the test set is very unbalanced classification accuracy is not a good measure of performance. You would be better off using precision/recall and the f-score, easily computed from the confusion matrix. Check this paper for references on classification performance measures for a lots of different scenarios.

Also another good read on the topic is this one.

Answer 2

Correct me if I am wrong, but the actual proportion of the classes in the population does not matter for ML in terms of the classification. Where it matters is in obtaining the training data for both classes that spans the entire feature space. So in general if you had a 80/20 split for classes A/B, it is much more likely that you are going to find an representative sample for class A relative to class B.

Thus On the question of whether its is better to have a 50/50 ratio, Intuitively my guess is that it might not be as important as having a larger training set by using all the data. However, this is the part that i am unsure of... which is how the unequal ratios could possibly impact misclassification rates, especially on the boundary of separation. For the test set, the proportions should not be relevant.