2
$\begingroup$

I'm working through a logistic regression example from the lab on logistic regression in Intro to Statistical Learning. When they try to test how accurate their model is they do,

glm.pred[glm.probs >.5] = "Up"

Essentially they are asking whether the predicted probability of a market increase is greater than or less than 0.5. But how did they choose the number 0.5? If there is another situation where probabilities are much lower for each prediciton, do we replace 0.5 with the mean(glm.probs)?

$\endgroup$
  • 1
    $\begingroup$ They discuss that a bit in the credit card default example. 0.5 makes sense in so far as it means the more likely of the two categories is predicted (when there are just two categories and the two probabilities have to add up to 1), but depending on the cost of getting the prediction wrong in each direction this may or may not be the most important thing. $\endgroup$ – Björn Apr 15 '16 at 19:42
  • $\begingroup$ @Björn Got it. So 0.5 makes sense because it is fairly even that the market will go up and down. But what if we have a situation where the probabilities of "up" are significantly lower (like avg = 0.02), and probability of "down" much higher (i.e. 0.98)? Is it ok to adjust the 0.5 'level' to 0.02? $\endgroup$ – jchaykow Apr 15 '16 at 19:45
2
$\begingroup$

It depends on the problem you are trying to solve. There are four important rates that one should look at for a problem: True Positive Rate, True Negative Rate, False Positive Rate and False Negative Rate. Changing the probability threshold will typically change each of these rates.

For example, let us consider the email spam detection problem. It may be more important that you get all your legitimate emails and are willing to accept a few emails that are spam. So, if your logistic regression model is predicting the probability of spam then you may want to set the probability higher than 0.5, say 0.9. This means that you want to classify an email as spam if your model's prediction of spam is at least 0.9.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ So the 'result' of the test is really the value that gives most accuracy? i.e. If we get 98% prediction accuracy with a probability 'level' of 0.9, then we can conclude that given the model predicts a 0.9 probability or higher, we can predict with highest accuracy. $\endgroup$ – jchaykow Apr 15 '16 at 19:49
  • 1
    $\begingroup$ It depends on the situation. As I mentioned there are typically trade-offs. If you want a high true positive rate you will typically end up increasing your false negative rate. If you want a high true negative rate you will typically end up increasing your false positive rate. You could associate a different cost with each of the rates and then choose the parameter value which minimizes your cost. $\endgroup$ – shoda Apr 15 '16 at 20:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.