1
$\begingroup$

In Logistic Regression, what's the logic of testing for all threshold probabilities in an ROC Plot?

I thought that in logistic regression, we're concerned with the 0.5 probability threshold, and if something is greater, we mark the outcome as 1, and if less, we mark the outcome as 0.

So why test all the possible threshold probabilities? Why not just test the case for the one we're concerned with?

$\endgroup$
  • 2
    $\begingroup$ You might find this helpful. $\endgroup$ – GeoMatt22 Nov 1 '16 at 0:26
  • 1
    $\begingroup$ You have to vary the cutoff to generate the ROC-AUC. And when you vary the cutoff, it's not simply 0.5. $\endgroup$ – JoleT Nov 1 '16 at 0:42
  • $\begingroup$ @GeoMatt22 Thank you, that was indeed pertinent. Usually when doing Logistic Regression, I don't actually specify the threshold (e.g. in R). So how does the threshold get chosen? And what is it usually? (I thought it was 0.5 always.) $\endgroup$ – thanks_in_advance Nov 1 '16 at 2:24
  • 1
    $\begingroup$ I am no expert, but the key point IMO is that logistic regression is really solving a certain MLE problem for the conditional probabilities of the classes. As a byproduct it gives a "soft classification". But for the "hard classification" (i.e. 0 or 1) problem, different loss functions are possible. In this "machine learning" context, the threshold to go from "soft to hard" is treated as an additional free parameter. $\endgroup$ – GeoMatt22 Nov 1 '16 at 3:25
2
$\begingroup$

The cutoff is ultimately something you have to choose using domain knowledge as well as the costs of getting it wrong. You also might have a case where your decision space isn't the same as the event you're trying to predict. For instance, if you predict the probability of a person having a given disease is < 1%, don't do anything, if it's < 50% observe regularly and carefully watching for more signs, and if it's > 50% perform another, possibly more invasive test.

Of course, that's just an example, but the idea is that what you do with the probabilities is up to you. There is no cutoff chosen by the logistic regression and it doesn't even have to correspond to "declare this instance a 0 or a 1". And even if it does, if the cost of calling a 1 a 0 is ten times that of calling a 0 a 1, you would be better off calling 1's even if you thought there were only a 20% (or lower chance) of it being 1. This idea comes up in what is called "cost sensitive learning".

Also, since logistic regression is just giving you the log-odds ratio (or equivalently the probability) if you're just looking for a measure of how well it does on a problem, you might not even care what label an individual class gets. You could look at loss functions based only on the predicted conditional class probabilities, like log loss or a Brier score. The AUC curve is a tool to look at the tradeoff of sensitivity and specificity if you were to set the cutoff at different points. As that other answer says, if you want to optimize for getting the most true positives subject to an upper bound on false negatives you'd do one thing, whereas if you wanted as few false negatives as possible with a lower limit on the number of true positives you'd do another.

At the end of the day, there's no rule, and it comes down to your understanding of the problem and reasons for using the method.

Edit: As @GeoMatt22 wisely suggested, the "other answer" I'm referring to that he originally mentioned is: Obtaining predicted values (Y=1 or 0) from a logistic regression model fit

$\endgroup$
  • $\begingroup$ You are absolutely correct, sir. And I have added it $\endgroup$ – Josh Magarick Nov 1 '16 at 3:39
1
$\begingroup$

If you haven't got an explicit loss function, you can also think about it this way. Where the ROC plot is flat, decreasing the cutoff reduces the false positives without increasing the false negatives much. Where the ROC plot is steep, increasing the cutoff reduces the false negatives without increasing the false positives much. So there is a sweet spot wherever there is a "shoulder", i.e. a local extremum in the second differential. You have to smooth it first, or think about the span over which you estimate the slope, because when you zoom in on a ROC curve it is all steps.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.