7
$\begingroup$

My datasets have two classes A and B. The classes should be treated equally (there is no "active/inactive"). The datasets are unbalanced, sometimes A is more frequent, sometimes B is more frequent. Which performance measure should I use?

Accuracy makes no sense on unbalanced datasets. If I get it right, F-measure and AUC assume that there is a active class: F-measure ignores true negatives as it is the harmonic mean of precision and recall. AUC ignores true negatives and false negatives.

So what performance measure should I use? Is AUC(active=A) + AUC(active=B) / 2 a valid option?

CORRECTION:

Apparently, I missunderstood how AUC works. It does NOT ignore true negatives and false negatives. The ROC curves look different depending on which class is considered the active one, but AUC(active=A) = AUC(active=B).

Improve this question
$\endgroup$
11
  • $\begingroup$ Do you have a cost function ? $\endgroup$ – image_doctor Apr 15 '13 at 10:10
  • $\begingroup$ Not quite sure what you mean. My instances have equal weights. My classifier gives a probability with each prediction. Does this answer the question? $\endgroup$ – user954923 Apr 15 '13 at 10:16
  • $\begingroup$ Your instances may have equal weights, do your errors have equal weights ? If they do, then accuracy is a valid measure of performance. $\endgroup$ – image_doctor Apr 15 '13 at 10:25
  • $\begingroup$ Each error has equal weight. How does this make accuracy valid? Lets assume I would like to compare validation results on two datasets, one is largely unbalanced, one is not? $\endgroup$ – user954923 Apr 15 '13 at 10:32
  • $\begingroup$ I'm not sure I understand the question, the accuracy on one dataset, say breast cancer prediction, and the accuracy on another dataset, say horse racing, are related in what way? Or are you saying you are sampling the same underlying distribution and the samples have varying ratios of class A and B? $\endgroup$ – image_doctor Apr 15 '13 at 10:37
4
$\begingroup$

Have a look at the Matthews Correlation Coefficient

$$MCC = \frac{TP \cdot TN - FP \cdot FN}{\sqrt{ (TP + FP)(TP + FN)(TN + FP)(TN + FN) }}$$

I have seen it pretty often as performance metrix in classification of SNPs dataset. Have a look at this link as well, they discuss the difference between AUC and MCC

Otherwise you can just compute an average accuracy (average error rate), I have seen people using it in multiclass problems as well.

$$AAcc = \frac{1}{2} \bigg( \frac{TP}{TP + FN} + \frac{TN}{TN + FP} \bigg) $$

Usually it is used in authentication systems under the form of Half Total Error Rate. E.g. here they provided a statistical test for that.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.