Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
In Logistic Regression, what's the logic of testing for all threshold probabilities in an ROC Plot?
I thought that in logistic regression, we're concerned with the 0.5 probability threshold, and if something is greater, we mark the outcome as 1, and if less, we mark the outcome as 0.
So why test all the possible threshold probabilities? Why not just test the case for the one we're concerned with?
The cutoff is ultimately something you have to choose using domain knowledge as well as the costs of getting it wrong. You also might have a case where your decision space isn't the same as the event you're trying to predict. For instance, if you predict the probability of a person having a given disease is < 1%, don't do anything, if it's < 50% observe regularly and carefully watching for more signs, and if it's > 50% perform another, possibly more invasive test.
Of course, that's just an example, but the idea is that what you do with the probabilities is up to you. There is no cutoff chosen by the logistic regression and it doesn't even have to correspond to "declare this instance a 0 or a 1". And even if it does, if the cost of calling a 1 a 0 is ten times that of calling a 0 a 1, you would be better off calling 1's even if you thought there were only a 20% (or lower chance) of it being 1. This idea comes up in what is called "cost sensitive learning".
Also, since logistic regression is just giving you the log-odds ratio (or equivalently the probability) if you're just looking for a measure of how well it does on a problem, you might not even care what label an individual class gets. You could look at loss functions based only on the predicted conditional class probabilities, like log loss or a Brier score. The AUC curve is a tool to look at the tradeoff of sensitivity and specificity if you were to set the cutoff at different points. As that other answer says, if you want to optimize for getting the most true positives subject to an upper bound on false negatives you'd do one thing, whereas if you wanted as few false negatives as possible with a lower limit on the number of true positives you'd do another.
At the end of the day, there's no rule, and it comes down to your understanding of the problem and reasons for using the method.
Edit: As @GeoMatt22 wisely suggested, the "other answer" I'm referring to that he originally mentioned is: Obtaining predicted values (Y=1 or 0) from a logistic regression model fit
If you haven't got an explicit loss function, you can also think about it this way. Where the ROC plot is flat, decreasing the cutoff reduces the false positives without increasing the false negatives much. Where the ROC plot is steep, increasing the cutoff reduces the false negatives without increasing the false positives much. So there is a sweet spot wherever there is a "shoulder", i.e. a local extremum in the second differential. You have to smooth it first, or think about the span over which you estimate the slope, because when you zoom in on a ROC curve it is all steps.
