4
$\begingroup$

I'm wondering which quality measures are available for non-binary classifiers. I've read this article

https://en.wikipedia.org/wiki/Receiver_operating_characteristic

I understand, that the idea of Confusion matrix can be generalized easily. However, such metrics as sensitivity,specificity or precision seem to make sense only for binary classification problems.

So, my question is: are there any developed methods for evaluting the quality of non-binary classifiers?

Thank you.

Improve this question
$\endgroup$

2 Answers 2

Reset to default
4
$\begingroup$

I'm not sure whether you're still interested in this topic or not, but I became curious about it and have read a little on the subject. I've decided to share the following information to store the findings for myself and with hope that this might be helpful for others as well.

It seems that there are quite a number of quality measures, which can be applied to various models, non-binary classification included. Some of them are mentioned on this page and in this paper. Another interesting paper, criticizing AUC measure can be found here. Please note that my answer might be a little more general, as I haven't restricted my search to classification problems.

Improve this answer
$\endgroup$
3
$\begingroup$

Kappa Statistic can be used.

k = (Acc - AccRand) / (1 - AccRand)

k = 1 is ideal, k = 0 means that the classifier is not better than random.

Acc is the accuracy of the classifier measured as the number of correctly classified observations over the total number of observations.

AccRand is the accuracy of a classifier making predictions by chance. AccRand can be easily measured empirically: take the outputs of your classifier, randomly permute predictions over observations, measure the accuracy.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thank you. As far as I can see, Scott's Pi is more appreciate for these purposes. en.wikipedia.org/wiki/Scott%27s_Pi As far as I can see, Kappa Statistics has some element of randomness(the element appears when you permute answers of your classifier in order to obtain AccRand). At the same time, Scott's Pi has no element of randomness since the calculation of Pr(e) based of Marginal sums. $\endgroup$
    – drsealks
    Commented May 26, 2014 at 12:58
  • $\begingroup$ Scott's pi makes the assumption that annotators have the same distribution of responses. This means that the classifier should have the same prior probabilities of the classes as the the dataset. Many classifiers do enforse this constraint. For example, the true classes may be {a,a,b,c,c}, and predictions may be {a,c,b,c,c} - the prior probabilities of the classes are not the same, but the prediction accuracy is good. $\endgroup$
    – inzl
    Commented May 26, 2014 at 14:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.