1
$\begingroup$

I have the following evaluation metrics on the test set, after running 6 models for a binary classification problem:

  accuracy logloss   AUC
1   19%      0.45   0.54
2   67%      0.62   0.67
3   66%      0.63   0.68
4   67%      0.62   0.66
5   63%      0.61   0.66
6   65%      0.68   0.42

I have the following questions:

  • How can model 1 be the best in terms of log-loss (the log-loss is the closest to 0) since it performs the worst (in terms of accuracy). What does that mean?
  • How come does model 6 have lower AUC score than e.g. model 5, when model 6 has better accuracy. What does that mean?
  • Is there a way to say which of these 6 models is the best?
Improve this question
$\endgroup$
5
  • 1
    $\begingroup$ What is the prevalence of each class? What is the threshold used to choose a class when calculating accuracy, and is the same threshold used for all 6 models? Can we see a confusion matrix and a plot of the ROC curve for at least one of the models? How many examples in the test set? Was the exact same test used to evaluate all six models (as opposed to say, cross validation.) All I can say from the information provided is that the log loss reported is very high: a constant model which always predicts p=0.5 will get a log loss of 0.7. Model 6 has an AUC less than 0.5 which is also a red flag. $\endgroup$
    – olooney
    Oct 29, 2019 at 16:18
  • 2
    $\begingroup$ Accuracy, log-loss and AUC provide different values because they answer different questions. The correct model is the one which produces the best trade-off for your organization. We can't tell you what that trade-off is because we don't know what kinds of costs are involved and we don't know how to compare the severity of the different kinds of errors. See also: stats.stackexchange.com/questions/414349/… $\endgroup$
    – Sycorax ♦
    Oct 29, 2019 at 16:31
  • 1
    $\begingroup$ See also: stats.stackexchange.com/questions/362982/… $\endgroup$
    – Sycorax ♦
    Oct 29, 2019 at 16:32
  • 1
    $\begingroup$ At first your AUC results are very bad which suggest your model generate random outputs. You should be aware that AUC is operating on the likelihoods and the accuracy processes final ouput of a model. Please check a prior distribution of the class over test set. You can easy get high accuracy with low AUC if your model learnt only a prior answer (i.e. always return the same value). $\endgroup$
    – podludek
    Oct 30, 2019 at 11:32
  • $\begingroup$ The final sentence by @podludek refers to a situation where you have, say, $99$ instances of one category for every instance of the other category. A model could achieve $99\%$ accuracy, which looks impressive, but that is the baseline rate, the accuracy you get from always predicting membership in the majority category. In such a situation, it could be that the model really has no ability to distinguish one category from the other, leading to $AUC = 0.5$ and gets what looks like a high accuracy score just because it can do so without having to know much (naïvely guess the majory every time). $\endgroup$
    – Dave
    Apr 5 at 20:22

1 Answer 1

Reset to default
1
$\begingroup$

Accuracy is a measure of how well the predictions correspond with the true categories after you apply some rule to the raw output to convert those values into hard classifications. The most common of these rules is to take the category that has the highest predicted probability, which corresponds to assigning to category $0$ if the prediction is below $0.5$ and to category $1$ if the prediction is above $0.5$. (For the rare case of having a prediction of exactly $0.5$, perhaps randomize in some way. This could be a discussion of its own.)

(This gets messier if you do not predict on $[0,1]$, but the idea remains that you pick a threshold where predictions below are assigned to one category and predictions above are assigned to the other.)

AUC is strictly a measure of ability to discriminate between the categories and has no regard for calibration (if events predicted to happen with probability $p$ really do happen with probability $p$). In fact, dividing every prediction by two does not change the AUC, which I demonstrate below with the exact same code I used in an answer I posted a few minutes ago that might be of interest.

library(pROC)
set.seed(2023)
N <- 1000
p <- rbeta(N, 1, 1)
y <- rbinom(N, 1, p)
pROC::roc(y, p)$auc   # I get 0.8481
pROC::roc(y, p/2)$auc # Again, I get 0.8481

The p and p/2 cannot both be calibrated, unless p is always zero and the event never occurs.

Finally, the log-loss measures both prediction prediction and the ability of the model to discriminate between the categories.

Overall, these three statistics (accuracy, AUC, log-loss) measure totally different aspects of the model in different ways. There is no reason to expect the three to agree on the best model, and what constitutes the best model for your work will depend on what you value and why you bother to do any modeling at all. If you need a model that tends to make good decisions when you apply a threshold (such as in software that lacks a human in the loop), maybe optimizing accuracy will be good for you (though I would advise giving thought to whether or not a threshold of $0.5$ is appropriate for your task). If you need to put the predictions in the correct order but otherwise do not care what the predictions are, AUC could be your friend. If you value having accurate probabilities, then a strictly proper scoring rule like log-loss or Brier score could be your performance metric of interest.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.