# Which performance measure for unbalanced binary classification without an 'active' class?

My datasets have two classes A and B. The classes should be treated equally (there is no "active/inactive"). The datasets are unbalanced, sometimes A is more frequent, sometimes B is more frequent. Which performance measure should I use?

Accuracy makes no sense on unbalanced datasets. If I get it right, F-measure and AUC assume that there is a active class: F-measure ignores true negatives as it is the harmonic mean of precision and recall. AUC ignores true negatives and false negatives.

So what performance measure should I use? Is AUC(active=A) + AUC(active=B) / 2 a valid option?

CORRECTION:

Apparently, I missunderstood how AUC works. It does NOT ignore true negatives and false negatives. The ROC curves look different depending on which class is considered the active one, but AUC(active=A) = AUC(active=B).

• Do you have a cost function ? Apr 15, 2013 at 10:10
• Not quite sure what you mean. My instances have equal weights. My classifier gives a probability with each prediction. Does this answer the question? Apr 15, 2013 at 10:16
• Your instances may have equal weights, do your errors have equal weights ? If they do, then accuracy is a valid measure of performance. Apr 15, 2013 at 10:25
• Each error has equal weight. How does this make accuracy valid? Lets assume I would like to compare validation results on two datasets, one is largely unbalanced, one is not? Apr 15, 2013 at 10:32
• I'm not sure I understand the question, the accuracy on one dataset, say breast cancer prediction, and the accuracy on another dataset, say horse racing, are related in what way? Or are you saying you are sampling the same underlying distribution and the samples have varying ratios of class A and B? Apr 15, 2013 at 10:37

$$MCC = \frac{TP \cdot TN - FP \cdot FN}{\sqrt{ (TP + FP)(TP + FN)(TN + FP)(TN + FN) }}$$
$$AAcc = \frac{1}{2} \bigg( \frac{TP}{TP + FN} + \frac{TN}{TN + FP} \bigg)$$