# Which performance measure for unbalanced binary classification without an 'active' class?

My datasets have two classes A and B. The classes should be treated equally (there is no "active/inactive"). The datasets are unbalanced, sometimes A is more frequent, sometimes B is more frequent. Which performance measure should I use?

Accuracy makes no sense on unbalanced datasets. If I get it right, F-measure and AUC assume that there is a active class: F-measure ignores true negatives as it is the harmonic mean of precision and recall. AUC ignores true negatives and false negatives.

So what performance measure should I use? Is AUC(active=A) + AUC(active=B) / 2 a valid option?

CORRECTION:

Apparently, I missunderstood how AUC works. It does NOT ignore true negatives and false negatives. The ROC curves look different depending on which class is considered the active one, but AUC(active=A) = AUC(active=B).

• Do you have a cost function ? – image_doctor Apr 15 '13 at 10:10
• Not quite sure what you mean. My instances have equal weights. My classifier gives a probability with each prediction. Does this answer the question? – user954923 Apr 15 '13 at 10:16
• Your instances may have equal weights, do your errors have equal weights ? If they do, then accuracy is a valid measure of performance. – image_doctor Apr 15 '13 at 10:25
• Each error has equal weight. How does this make accuracy valid? Lets assume I would like to compare validation results on two datasets, one is largely unbalanced, one is not? – user954923 Apr 15 '13 at 10:32
• I'm not sure I understand the question, the accuracy on one dataset, say breast cancer prediction, and the accuracy on another dataset, say horse racing, are related in what way? Or are you saying you are sampling the same underlying distribution and the samples have varying ratios of class A and B? – image_doctor Apr 15 '13 at 10:37

$$MCC = \frac{TP \cdot TN - FP \cdot FN}{\sqrt{ (TP + FP)(TP + FN)(TN + FP)(TN + FN) }}$$
$$AAcc = \frac{1}{2} \bigg( \frac{TP}{TP + FN} + \frac{TN}{TN + FP} \bigg)$$