I am constructing a predictive model for a binary outcome, which gives me results in a "probability-prediction" fashion such as "0.3-YES" or "0.4-NO". The model is working perfectly.

My question here is what would be the probability that I should trust? is any value below 0.35 or above 0.65 confident enough?

I know this varies according to my data, but I am asking about the typically used value to consider my prediction as "confident". For example in hypothesis testing, a 0.05 p value is the most commonly used value for significance of the existence of a difference between populations. What is the typically used values for a probability that ranges between 0-1?

  • 1
    $\begingroup$ This isn't answering your question so is a comment... If it were me I would be considering two things (1) Prediction intervals: you should be able to get your model to output a prediction interval, e.g. [0.25, 0.35] rather than the point estimate 0.3 meaning that 95% of the time the interval will contain the true probability for your inputs. (2) which category to choose, eg, YES or NO. Here it is usual to simply choose the category for which your model outputs the highest probability. The prediction intervals should give you some idea as to how confident to be about categorization. $\endgroup$ – TooTone Aug 15 '13 at 22:19
  • $\begingroup$ Thanks for your comment @TooTone. This is exactly what I am currently doing, manual adjustment of the parameters of my prediction, and it is quite annoying because you are never too sure about the values (or ranges) that you are setting to be the cut value for your acceptance/rejection. I was hoping to find a "common" value just like the 0.05 p value of the hypothesis testing, seems like there is no such value. I'll keep on looking anyway. Cheers $\endgroup$ – Error404 Aug 16 '13 at 11:33
  • $\begingroup$ I think the answer depends on whether you care more about false positive or false negative results. By choosing a probability threshold you can balance the model's precision and recall in a way that suits your use case. $\endgroup$ – gereleth Apr 22 '17 at 9:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.