13
$\begingroup$

I have a binary classification problem and I experiment different classifiers on it: I want to compare the classifiers. which one is a better measure AUC or accuracy? And why?

Raondom Forest: AUC: 0.828  Accuracy: 79.6667 %
           SVM: AUC: 0.542  Accuracy: 85.6667 %
Improve this question
$\endgroup$

3 Answers 3

Reset to default
15
$\begingroup$

Proportion classified correctly is an improper scoring rule, i.e., it is optimized by a bogus model. I would use the quadratic proper scoring rule known as the Brier score, or the concordance probability (area under ROC curve in the binary $Y$ case). Random forest works better than SVM in your case.

Improve this answer
$\endgroup$
2
  • $\begingroup$ If for subject $i$ in your sample $o_i \in \{0,1\}$ is the observed binary outcome and $\hat{f}_i$ is the predicted probability of a '1' then the Brier score is (if I remember) $B=\frac{1}{n} \sum_{i=1}^n (\hat{f}_i - o_i)^2$. As the OP has a binary classification problem the $o_i$ are known but how do you compute $\hat{f}_i$ for SVM ? $\endgroup$
    – user83346
    Commented Aug 5, 2016 at 10:04
  • $\begingroup$ @fcop There is a way to transform a SVM's binary classification prediction into a probability, called Platt Scaling (en.wikipedia.org/wiki/Platt_scaling). Essentially, rather than computing the SVM classification $\hat y_i$ ($= +1$ or $-1$) as $\hat y_i = sign(g(y_i,x_i))$, where $g(y_i,x_i)$ is the solution to the SVM convex quadratic programming problem, Platt scaling takes a logistic transformation of $g(y_i,x_i)$: $\hat f_i = P(Y=1|x_i)=\frac{1}{1+exp(A \times g(y_i,x_i) + B)}$ where $A$ and $B$ are parameters determined by the Platt scaling algorithm. $\endgroup$
    – RobertF
    Commented Aug 26, 2016 at 15:54
9
$\begingroup$

I think you should definitely look into more metrics than just AUC and accuracy.

Accuracy (together with sensitivity and specificity) is a very simple but biased metric which forces you to look at the absolute prediction result and does not open for assertion of class probabilities or ranking. It also does not take the population into account which invites to misinterpretation as a model giving a 95% accuracy on a population with 95% chance of being correct at random isn't really a good model, even if the accuracy is high.

AUC is a good metric for asserting model accuracy that is independent of population class probabilities. It will, however not tell you anything about how good the probability estimates actually are. You could get a high AUC but still have very skewed probability estimates. This metric more discriminating than accuracy and will definitely give you better models when used in combination with some proper scoring rule, e.g Brier score as mentioned in another post.

You can get a more formal proof here, although this paper is quite theoretical: AUC: a Statistically Consistent and more Discriminating Measure than Accuracy

There are however a bunch of good metrics available. Loss Functions for Binary Class Probability Estimation and Classification: Structure and Applications is a good paper investigaing proper scoring rules such as the Brier score.

Another interesting paper with metrics for assertion of model performance is Evaluation: from precision, recall and F-measure to ROC, informedness, markedness & correlation taking up other good performance metrics such as informedness.

To summarize I would recommend looking at AUC/Gini and Brier score to assert you model performance, but depending on the goal with your model other metrics might suit your problem better.

Improve this answer
$\endgroup$
3
  • $\begingroup$ The link for Evaluation: from precision, recall and F-measure to ROC, informedness, markedness & correlation is dead $\endgroup$
    – vonjd
    Commented May 24, 2016 at 14:55
  • $\begingroup$ If for subject $i$ in your sample $o_i \in \{0,1\}$ is the observed binary outcome and $\hat{f}_i$ is the predicted probability of a '1' then the Brier score is (if I remember) $B=\frac{1}{n} \sum_{i=1}^n (\hat{f}_i - o_i)^2$. As the OP has a binary classification problem the $o_i$ are known but how do you compute $\hat{f}_i$ for SVM ? $\endgroup$
    – user83346
    Commented Aug 5, 2016 at 10:05
  • $\begingroup$ No brierscore is not great for methods that only gives you an outcome and not a probability. Niether is auc though as this will tell you how well you rank your predictions. With only outcomes you'll only get a point in ROC space hence giving you the area under the curve will be the triangle. But it'll still give you a number and so will brierscore allthough it will more or less transform into 0-1 loss. If you have only outcomes i suggest looking at Precision, Recall and Cohen's Kappa which are metrics designed for when you have outcomes. $\endgroup$
    – while
    Commented Aug 10, 2016 at 9:39
2
$\begingroup$

With respectful deference to Dr. Harrell’s answer regarding proper scoring metrics like the Brier Score, if only the two choices of Accuracy and AUC ROC are given, the answer is it depends upon the data and the desired outcome measure.

• The Data: AUC ROC is prevalence-invariant; it will not vary from class imbalance. If your binary classification dataset is not balanced (nearly equal positive and negative examples), you won’t know from the AUC. Accuracy is a poor metric in the imbalanced case.

• The Question: If only concerned with correct prediction, accuracy is fine. If the real-word penalty for a ‘miss’ is similar to a ‘hit’, accuracy is fine. However, accuracy does not distinguish between errors and can overestimate the algorithm’s ability. AUC ROC gives a good comparison of two models, but is only a starting point as it represents ALL potential operating points, not the SINGLE operating point an algorithm would function at. Two algorithms with the same AUC ROC can each be superior at different operating thresholds if they do not dominate each other.

The Powers Paper Evaluation: From Precision, Recall, and F-Factor…is helpful. Updated link.

So, both are inadequate. However, in a balanced dataset, between two algorithms tested on the same data, AUC ROC is probably a better measure than accuracy.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.