3
$\begingroup$

I'm working through a logistic regression example from the lab on logistic regression in Intro to Statistical Learning. When they try to test how accurate their model is they do,

glm.pred[glm.probs >.5] = "Up"

Essentially they are asking whether the predicted probability of a market increase is greater than or less than 0.5. But how did they choose the number 0.5? If there is another situation where probabilities are much lower for each prediciton, do we replace 0.5 with the mean(glm.probs)?

Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ They discuss that a bit in the credit card default example. 0.5 makes sense in so far as it means the more likely of the two categories is predicted (when there are just two categories and the two probabilities have to add up to 1), but depending on the cost of getting the prediction wrong in each direction this may or may not be the most important thing. $\endgroup$
    – Björn
    Commented Apr 15, 2016 at 19:42
  • $\begingroup$ @Björn Got it. So 0.5 makes sense because it is fairly even that the market will go up and down. But what if we have a situation where the probabilities of "up" are significantly lower (like avg = 0.02), and probability of "down" much higher (i.e. 0.98)? Is it ok to adjust the 0.5 'level' to 0.02? $\endgroup$
    – conv3d
    Commented Apr 15, 2016 at 19:45

1 Answer 1

Reset to default
2
$\begingroup$

It depends on the problem you are trying to solve. There are four important rates that one should look at for a problem: True Positive Rate, True Negative Rate, False Positive Rate and False Negative Rate. Changing the probability threshold will typically change each of these rates.

For example, let us consider the email spam detection problem. It may be more important that you get all your legitimate emails and are willing to accept a few emails that are spam. So, if your logistic regression model is predicting the probability of spam then you may want to set the probability higher than 0.5, say 0.9. This means that you want to classify an email as spam if your model's prediction of spam is at least 0.9.

Improve this answer
$\endgroup$
2
  • $\begingroup$ So the 'result' of the test is really the value that gives most accuracy? i.e. If we get 98% prediction accuracy with a probability 'level' of 0.9, then we can conclude that given the model predicts a 0.9 probability or higher, we can predict with highest accuracy. $\endgroup$
    – conv3d
    Commented Apr 15, 2016 at 19:49
  • 1
    $\begingroup$ It depends on the situation. As I mentioned there are typically trade-offs. If you want a high true positive rate you will typically end up increasing your false negative rate. If you want a high true negative rate you will typically end up increasing your false positive rate. You could associate a different cost with each of the rates and then choose the parameter value which minimizes your cost. $\endgroup$
    – shoda
    Commented Apr 15, 2016 at 20:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.