Let's say I choose threshold in logistic regression equals to $0.8$. If it is lower then class is $0$ else is $1$. Then, how do I interpret the outcome on test point $x$, $h(x)=0.6$, where $h$ is my logistic regression model. I think I cannot say that with probability equals $0.6$ class of $x$ is $1$? Is there any way to interpret this? Should I just scale appropriate intervals so in that case the outcome $0.6$ I can interpret as probability of $0.375$ that point is in class $1$?

  • 1
    $\begingroup$ Since logistic regression does not use any threshold by itself & using threshold for making hard classifications has no impact on the probabilities predicted by logistic regression, what exactly do you mean? $\endgroup$ – Tim Jun 15 at 16:09
  • $\begingroup$ Yes. I understand that if I use threshold $0.5$ (usually it is default) then everything can be interpreted nicely. But how do I interpret things when I do not use $0.5$? $\endgroup$ – amad Jun 15 at 16:28

You have to be careful to distinguish the probability estimate from a logistic regression model from the probability cutoff you use when you apply the model in practice to make a decision.

Logistic regression results can be expressed in terms of the probability of class membership. I take your terminology $h(x) = 0.6$ to mean that your model $h$ predicts that case $x$ has probability of 0.6 of belonging to Class 1. That's the probability estimate from the model. It has nothing to do directly with a probability cutoff.

When you apply the probability model, practical considerations come into play. Say that false-positive decisions about membership in Class 1 cost more than do false-negative decisions. Then you would want to avoid a decision that places a true Class 0 case into Class 1. Using a high probability cutoff like 0.8 to make assignments of class membership for your purpose would accomplish that.

So if a case has $h(x) = 0.6$ that still means it has a 60% chance of actually being in Class 1. You just will have made a decision not to assign it to Class 1 for your purposes because you don't want to risk that it really is in Class 0.

| cite | improve this answer | |
  • $\begingroup$ Thanks. If I choose threshold $0.6$ because it minimizes error and gives the best predictions then it means that my model is not correct? Am I right that on the training set error is minimized for the threshold $0.5$ and changing threshold may minimize error only on the test set? $\endgroup$ – amad Jun 15 at 16:33
  • $\begingroup$ @amad it depends on what you mean by "best predictions." "Training error" in terms of fraction of cases correctly assigned is not a good choice for determining "best predictions." A logistic regression model minimizes overall log loss, not necessarily the fraction of cases correctly assigned if you choose a probability cutoff of 0.5. The "best predictions" in practice that are based on a cutoff for a decision rule depend on the relative costs of false-negative and false-positive assignments to Class 1. $\endgroup$ – EdM Jun 15 at 16:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.