2
$\begingroup$

I am working with a highly class-skewed three class classification problem. The class percentages are A = 1.8%, B = 17.5% and C = 80.7%. According to this paper, the following definition of multi-class AUC is insensitive to class distributions and therefore, I am using it:enter image description here

Now I am testing 10 different classfiers on the dataset and only one of them has $AUC_{total} > 0.5$. Am I right in assuming that $AUC_{total} = 0.5$ for a random classifier? If yes, are 9 of the classifiers worse than a random classifier (on this data) just because they have a lower $AUC_{total}$? If yes again, can you suggest possible reasons for such poor performance and what can be done.

P.S.: Even the best performing classifier has $AUC_{total}$ of about 0.6.

$\endgroup$
  • $\begingroup$ Elvis has a good answer, but I would add (1) have you tried the classification model that predicts C for all cases? This seems like the better baseline prediction model. (2) Using the c-statistic to compare models can be suboptimal. There are many threads on this site discussing this issue. $\endgroup$ – charles Nov 18 '13 at 1:34
1
$\begingroup$

I assume that to compute $AUC(c_i,c_j)$, you use the classifier obtained by comparing the probabilities of pertaining to these two classes.

Then, for the random classifier which draws uniformly a point $(p_1, \dots, p_n)$ in the $n$-simplex (I denote $n = |C|$), you have $AUC(c_i, c_j) = 0.5$.

As there are $n( n - 1)$ pairs $(c_i, c_j)$, if each $AUC(c_i, c_j) = 0.5$ then $AUC_{total} = 0.5$, yes.

$\endgroup$
  • $\begingroup$ Actually, no. I used the multi-class variants of all 10 classifiers. Since not all classifiers give probabilities (like SVM), I treated all 10 as 'discrete' classifiers. As you may know, discrete classifiers only give a single point on the ROC. In the paper that I linked in the OP, the author recommends calculating $AUC$ for discrete classifiers by connecting this single point to $(0,0)$ and $(1,1)$ and taking the area enclosed. I did the same. For three classes I get 3 ROCs (AB, BC and AC) and hence 3 AUCs for each classifier. $AUC_total$ is simply the average of the three. $\endgroup$ – Prometheus Nov 18 '13 at 9:49
  • $\begingroup$ Note that the classifier I described (predict the class with maximum probability) is discrete. The point is that you have to build two-class classifiers from each multi-class classifier you consider. When you compute $AUC(A,B)$, what do you do when the classifier predicts "class C"? $\endgroup$ – Elvis Nov 18 '13 at 11:10
  • $\begingroup$ For $AUC(A,B)$, I simply disregard all labels with class $C$. In other words, if $A$ is "positive" and $B$ is "negative", then $FPR = \frac{\mbox{No. of B classified as A}}{\mbox{No. of B}}$ and $TPR=\frac{\mbox{No. of A classified as A}}{\mbox{No. of A}}$. $\endgroup$ – Prometheus Nov 18 '13 at 11:30
  • $\begingroup$ No wonder you have $AUC < 0.5$ then... to make the AUC make sense, you have to define a binary classifier. You can’t have ``unclassified'' elements. $\endgroup$ – Elvis Nov 18 '13 at 13:36
  • $\begingroup$ But then how do I calculate $AUC(A,B)$ without disregarding class $C$ ? $\endgroup$ – Prometheus Nov 18 '13 at 15:46
0
$\begingroup$

Most classifiers are notoriously bad at unbalanced problems because they optimize error, accuracy or f-measure which is basically saying all INSTANCES are equal. On the other hand measures like ROC AUC, Gini, Informedness, Correlation and Kappa are designed on the assumption that all CLASSES are equal. Another possibility is that you COST classes or instances explicitly and use that as a basis for optimization.

In your case, saying everything is C will net you an accuracy of 80.7 and most neural network and tree classifiers have a bias to this for such an unbalanced set. However, the chance-correct evaluation of this is 0 (Gini=Youden=Informedness=Correlation=Kappa=0 means chance level performance is obtained). Any model which allocates A% to A, B% to B, C% to C randomly should average out (over multiple runs) to 0 for these measures. The "say everything is C" model arbitrarily allocates 0% to A, 0% to B, 100% to C (randomly with the indicated probability).

You need to look for algorithms that optimize the measure you are interested in, viz. ROC AUC or Gini or Informedness (chance-correct learning), or explicitly balance (balanced learning), or allow you to specify class or instance costs explicitly(cost-sensitive learning). On the other hand, some algorithms explicitly debalance (nodes in a decision tree or stages of boosting for example).

I've written several papers about these issues, including deriving chance-correct versions of both NN and Boosting algorithms. There are a lot of people using Kappa and AUC - but you need to look particularly at those addressing the multiclass case. And there is a growing literature on balanced/unbalanced/rebalanced learning.

$\endgroup$
  • 1
    $\begingroup$ "Most classifiers are notoriously bad at unbalanced problems because they optimize error, accuracy or f-measure which is basically saying all INSTANCES are equal." Which classifiers are these? $\endgroup$ – Reinstate Monica Mar 29 '16 at 13:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.