What are some possible classification metric for an unbalanced problem ? Due to skeweness of the distribution, accuracy value is not so meaningful. For instance, if I predict all the classes to class 1 I could still get 70% accuracy.

  • $\begingroup$ What about the average value of the diagonal entries of the confusion matrix? $\endgroup$ – Sobi Dec 13 '15 at 0:19
  • $\begingroup$ This doesn't incorporate class imbalance. $\endgroup$ – Archie Jan 19 '17 at 9:12

My apologies, just saw how old the question was -- why was it on the top of the list?

Answer (which is as good as it gets with limited information):

Of what kind is the data?

You should probably never use detection accuracy or certainly not when your classifier outputs a score or probability. How do you classify? The underlying loss function of your classification algorithm is usually a good measure to start with when it comes to evaluation performance.

I would not lean towards 1~vs~all analytic approaches, such as the precision recall curve(s). It won't get you very far -- you would have to test each class against all others and then combine these results somehow. Harmonic mean, a-priori likelihood given the class to be tested, ... ? It is unclear what these measures will actually tell you.

If you have probabilistic output , the negative log likelihood is a good place to start with.

If you already have 70% accuracy for class 1, which means 70% of your dataset are class 1, then you might be in the situation that your classifier gives up on some smaller classes and rather tries to satisfy a possible regularization term. But this is all really dependent on your classification scheme. If you want a clearer answer, you need to tell us the whole story. ;)

  • 1
    $\begingroup$ Don't worry about how old the question is. This site tries to build a repository of good answers to questions, to last beyond the immediate needs of the original questioner. The system dredges up old questions to the top of the queue about once an hour under the "community" user. If that's what happened in this case it served its purpose well, as your answer is much better, in my opinion, to the one posted 9 months ago. $\endgroup$ – EdM Feb 13 '16 at 14:40
  • $\begingroup$ Would you recommend some form of weighted AUC? $\endgroup$ – Archie Jan 19 '17 at 9:12

Try the F1-score, which balances precision and recall.

Precision can be calculated by the number of true positives divided by total positives, and recall by the number of true positives divided by the total number of elements that actually belong to the positive class. These are weighted by a harmonic mean.

  • 1
    $\begingroup$ How about for multi-class classification problem? Should we calculate one F1-score per class $\endgroup$ – Seeda May 17 '15 at 15:16
  • $\begingroup$ You should try micro f1 for unbalanced multiclass. $\endgroup$ – chandresh May 16 at 15:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.