First of all, I would like to describe some common layouts that Data Mining books use explaining how to deal with Unbalanced Datasets. Usually the main section is named Unbalanced Datasets and they cover these two subsections: Cost-Sensitive Classification and Sampling Techniques.

It seems that facing a problem with a rare class you can perform both cost-sensitive classification and sampling. Instead, I think that one should apply cost-sensitive techniques if the rare class is also the target of the classification and a misclassification of a record of that class is costly.

On the other hand, sampling techniques, such as over-sampling and under-sampling, are useful if the target of the classification is a good accuracy overall, without focusing on a particular class.

This belief comes from the rationale of MetaCost that is a general way to make a classifier cost-sensitive: if one want to make a classifier cost-sensitive in order to penalise a misclassification error of the rare class he should over-sample the other class. Roughly speaking, the classifier tries to adapt to the other class and it becomes specific to the rare class.

This is the opposite of over-sampling the rare class, that is the usually suggested way to deal with this problem. Over-sampling of the rare class or the under-sampling of the other class is useful to improve the overall accuracy.

Please, it would be great if you confirmed my thoughts.

Stated this, the common question facing an unbalanced dataset is:

Should I try to get a dataset that counts as many rare records as other ones?

My answer would be, in case you are looking for accuracy: OK. You can perform it either finding out more rare class examples or deleting some records of the other class.

In case you are focusing on the rare class, with a cost-sensitive technique, I would answer: you can only find out more rare class example but you shouldn't delete records of the other class. In the latter case you will not be able to let the classifier adapt to the other class, and the rare class misclassification error could increase.

What would you answer?

  • 2
    $\begingroup$ "Finding out" new records for rare classes might be impossible. I suppose the data is structured this way because it is expensive (bioinformatics) or risky (bank loan) to create more rare-class-events. $\endgroup$ – steffen Oct 19 '11 at 7:10
  • $\begingroup$ Of course, but it is a common proposed solution. However, it is true that if you can find out more rare class examples you might be able to find out also other examples. Because the training set should be a representative sample of the record universe. So, it seems to me it looks like performing over-sampling. $\endgroup$ – Simone Oct 19 '11 at 8:06

It's a good question. Personally, my answer would be that it never makes sense to throw data away (unless it is for computational reasons), as the more data you have, the better your model of the world can be. Therefore, I would suggest that modifying the cost function in appropriate way for your task should be sufficient. For example, if you are interested in one particular rare class, you can make misclassifications of this class only more expensive; if you are interested in a balanced measure, something like Balanced Error Rate (the average of the errors on each class) or the Matthews Correlation Coefficient is appropriate; if you are interested only in overall classification error, the traditional 0-1 loss.

A modern approach to the problem is to use Active Learning. For example, Hospedales et al (2011) "Finding Rare Classes: Active Learning with Generative and Discriminative Models, IEEE Transactions on Knowledge and Data Engineering, (TKDE 2011). However I believe the these approaches are still relatively less mature.

  • $\begingroup$ Interesting measure the Metthews one in case one needed a balanced measure. However, given that we don't want to delete any record, prior to perform any sampling or modification to the cost function, would you rebalance the dataset adding rare class examples? I think the answer could be NO. Because as long as you find out rare class examples you can find out other examples. Thus, to get a better balanced measure or a better rare class performance measure (e.g. F-measure) I would perform a technique (such as sampling or mod of the cost) only after the data collection phase. Do you agree? $\endgroup$ – Simone Dec 8 '11 at 13:59
  • $\begingroup$ Agreed, any operations such as this should be performed after the data collection phase. $\endgroup$ – tdc Dec 8 '11 at 14:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.