Let's say I choose threshold in logistic regression equals to $0.8$. If it is lower then class is $0$ else is $1$. Then, how do I interpret the outcome on test point $x$, $h(x)=0.6$, where $h$ is my logistic regression model. I think I cannot say that with probability equals $0.6$ class of $x$ is $1$? Is there any way to interpret this? Should I just scale appropriate intervals so in that case the outcome $0.6$ I can interpret as probability of $0.375$ that point is in class $1$?
2
-
1$\begingroup$ Since logistic regression does not use any threshold by itself & using threshold for making hard classifications has no impact on the probabilities predicted by logistic regression, what exactly do you mean? $\endgroup$– TimCommented Jun 15, 2020 at 16:09
-
$\begingroup$ Yes. I understand that if I use threshold $0.5$ (usually it is default) then everything can be interpreted nicely. But how do I interpret things when I do not use $0.5$? $\endgroup$– amadCommented Jun 15, 2020 at 16:28
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
2
You have to be careful to distinguish the probability estimate from a logistic regression model from the probability cutoff you use when you apply the model in practice to make a decision.
Logistic regression results can be expressed in terms of the probability of class membership. I take your terminology $h(x) = 0.6$ to mean that your model $h$ predicts that case $x$ has probability of 0.6 of belonging to Class 1. That's the probability estimate from the model. It has nothing to do directly with a probability cutoff.
When you apply the probability model, practical considerations come into play. Say that false-positive decisions about membership in Class 1 cost more than do false-negative decisions. Then you would want to avoid a decision that places a true Class 0 case into Class 1. Using a high probability cutoff like 0.8 to make assignments of class membership for your purpose would accomplish that.
So if a case has $h(x) = 0.6$ that still means it has a 60% chance of actually being in Class 1. You just will have made a decision not to assign it to Class 1 for your purposes because you don't want to risk that it really is in Class 0.
-
$\begingroup$ Thanks. If I choose threshold $0.6$ because it minimizes error and gives the best predictions then it means that my model is not correct? Am I right that on the training set error is minimized for the threshold $0.5$ and changing threshold may minimize error only on the test set? $\endgroup$– amadCommented Jun 15, 2020 at 16:33
-
$\begingroup$ @amad it depends on what you mean by "best predictions." "Training error" in terms of fraction of cases correctly assigned is not a good choice for determining "best predictions." A logistic regression model minimizes overall log loss, not necessarily the fraction of cases correctly assigned if you choose a probability cutoff of 0.5. The "best predictions" in practice that are based on a cutoff for a decision rule depend on the relative costs of false-negative and false-positive assignments to Class 1. $\endgroup$– EdMCommented Jun 15, 2020 at 16:52