11
$\begingroup$

I have a binary classification problem and I experiment different classifiers on it: I want to compare the classifiers. which one is a better measure AUC or accuracy? And why?

Raondom Forest: AUC: 0.828  Accuracy: 79.6667 %
           SVM: AUC: 0.542  Accuracy: 85.6667 %
$\endgroup$
13
$\begingroup$

Proportion classified correctly is an improper scoring rule, i.e., it is optimized by a bogus model. I would use the quadratic proper scoring rule known as the Brier score, or the concordance probability (area under ROC curve in the binary $Y$ case). Random forest works better than SVM in your case.

$\endgroup$
  • $\begingroup$ If for subject $i$ in your sample $o_i \in \{0,1\}$ is the observed binary outcome and $\hat{f}_i$ is the predicted probability of a '1' then the Brier score is (if I remember) $B=\frac{1}{n} \sum_{i=1}^n (\hat{f}_i - o_i)^2$. As the OP has a binary classification problem the $o_i$ are known but how do you compute $\hat{f}_i$ for SVM ? $\endgroup$ – user83346 Aug 5 '16 at 10:04
  • $\begingroup$ @fcop There is a way to transform a SVM's binary classification prediction into a probability, called Platt Scaling (en.wikipedia.org/wiki/Platt_scaling). Essentially, rather than computing the SVM classification $\hat y_i$ ($= +1$ or $-1$) as $\hat y_i = sign(g(y_i,x_i))$, where $g(y_i,x_i)$ is the solution to the SVM convex quadratic programming problem, Platt scaling takes a logistic transformation of $g(y_i,x_i)$: $\hat f_i = P(Y=1|x_i)=\frac{1}{1+exp(A \times g(y_i,x_i) + B)}$ where $A$ and $B$ are parameters determined by the Platt scaling algorithm. $\endgroup$ – RobertF Aug 26 '16 at 15:54
8
$\begingroup$

I think you should definitely look into more metrics than just AUC and accuracy.

Accuracy (together with sensitivity and specificity) is a very simple but biased metric which forces you to look at the absolute prediction result and does not open for assertion of class probabilities or ranking. It also does not take the population into account which invites to misinterpretation as a model giving a 95% accuracy on a population with 95% chance of being correct at random isn't really a good model, even if the accuracy is high.

AUC is a good metric for asserting model accuracy that is independent of population class probabilities. It will, however not tell you anything about how good the probability estimates actually are. You could get a high AUC but still have very skewed probability estimates. This metric more discriminating than accuracy and will definitely give you better models when used in combination with some proper scoring rule, e.g Brier score as mentioned in another post.

You can get a more formal proof here, although this paper is quite theoretical: AUC: a Statistically Consistent and more Discriminating Measure than Accuracy

There are however a bunch of good metrics available. Loss Functions for Binary Class Probability Estimation and Classification: Structure and Applications is a good paper investigaing proper scoring rules such as the Brier score.

Another interesting paper with metrics for assertion of model performance is Evaluation: from precision, recall and F-measure to ROC, informedness, markedness & correlation taking up other good performance metrics such as informedness.

To summarize I would recommend looking at AUC/Gini and Brier score to assert you model performance, but depending on the goal with your model other metrics might suit your problem better.

$\endgroup$
  • $\begingroup$ The link for Evaluation: from precision, recall and F-measure to ROC, informedness, markedness & correlation is dead $\endgroup$ – vonjd May 24 '16 at 14:55
  • $\begingroup$ If for subject $i$ in your sample $o_i \in \{0,1\}$ is the observed binary outcome and $\hat{f}_i$ is the predicted probability of a '1' then the Brier score is (if I remember) $B=\frac{1}{n} \sum_{i=1}^n (\hat{f}_i - o_i)^2$. As the OP has a binary classification problem the $o_i$ are known but how do you compute $\hat{f}_i$ for SVM ? $\endgroup$ – user83346 Aug 5 '16 at 10:05
  • $\begingroup$ No brierscore is not great for methods that only gives you an outcome and not a probability. Niether is auc though as this will tell you how well you rank your predictions. With only outcomes you'll only get a point in ROC space hence giving you the area under the curve will be the triangle. But it'll still give you a number and so will brierscore allthough it will more or less transform into 0-1 loss. If you have only outcomes i suggest looking at Precision, Recall and Cohen's Kappa which are metrics designed for when you have outcomes. $\endgroup$ – while Aug 10 '16 at 9:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.