4
$\begingroup$

I can't quite wrap my head around something that should be relatively fundamental. I'm familiar with ROC curves and AUC, but what I'm confused about is using accuracy for logistic regression. My understanding is that a logistic regression model outputs probabilities, and then those probabilities can be classified as one of either class (assuming a binary dependent variable), based on a threshold value. This makes sense in the case of an AUC value, since the value represents the entire range of the threshold from 0 to 1. But how can we use "accuracy" as a way to determine the performance of a model when the predicted classes are dependent on the threshold value? I keep seeing it used in examples of Python code (using sklearn) but don't understand how it works. Using AUC make sense, but accuracy I don't get.

Shouldn't we get a whole range of accuracy values for each model, the same way we get a whole range of true positive and false positive rates for each model that combine to give a single AUC score?

Improve this question
$\endgroup$
2
  • 2
    $\begingroup$ Its a bad practice and you are correct to be suspicious. Many of these sources indiscriminately apply a threshold of 0.5, which has many issues. $\endgroup$
    – Matthew Drury
    May 18, 2017 at 0:04
  • $\begingroup$ Why not simply call it as you see it and refer to the AUC as the AUC? "Accuracy" is too ambiguous: technically a non-deterministic model is inaccurate since it can't predict the outcome correctly with probability 1. If anything: the AUC is a measure of recall, if you use its proper definition. $\endgroup$
    – AdamO
    Jun 2, 2017 at 15:46

1 Answer 1

Reset to default
3
$\begingroup$

As @MatthewDrury writes, this really isn't something people should typically be doing. People do it, to be blunt, because they don't know what they're doing and they are mechanically mimicking what they have seen others do. For more information about this topic, see my answer here: RMSE (Root Mean Squared Error) for logistic models.

To answer your explicit question, the accuracy of a model's classifications are contingent on the model plus the threshold. A better model, or the same model with a different threshold, may have better accuracy. In essence, people are saying 'here is the accuracy we will get if we use this model with this threshold'.

Improve this answer
$\endgroup$
4
  • $\begingroup$ If a different threshold gave a better accuracy, wouldn't that imply that the logistic regression model gave badly calibrated estimates of the posterior probability of class membership? $\endgroup$
    – Dikran Marsupial
    Mar 23, 2022 at 12:09
  • 1
    $\begingroup$ @DikranMarsupial, not necessarily. In a medical context, eg, people often want to identify the small minority of patients who w/ a problem. You maximize accuracy by using .5, but such a model can be worthless, by not identifying anyone. It's common for people to say 'I care so much about false positives & so much about false negatives', or 'I want to identify any patient whose risk is >X'. In either case, we can determine a corresponding threshold, & that will correspond to some accuracy, but a different threshold (ie .5) would yield better accuracy, even though the calibration is fine. $\endgroup$
    – gung - Reinstate Monica
    Mar 23, 2022 at 13:34
  • $\begingroup$ "correspond to some accuracy," in that case, the expected loss would be the more appropriate metric. I personally would always take accuracy to be a statement about performance for equal misclassification costs. I think accuracy is a reasonable performance statistic for logistic regression, as long as it was understood that it was about the calibration of one particular probability contour. If it is the contour relevant to your application it may be useful. I certainly wouldn't use it as the only metric though! $\endgroup$
    – Dikran Marsupial
    Mar 23, 2022 at 13:50
  • $\begingroup$ I fully agree with "People do it, to be blunt, because they don't know what they're doing and they are mechanically mimicking what they have seen others do. For more information about this topic" - this is one of the problems with blogs etc. with code snippets. $\endgroup$
    – Dikran Marsupial
    Mar 23, 2022 at 13:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.