I am interested in finding out how confident my model (say Logistic Regression) is in predicting the label of a new data point. For example, if it is not confident, I better abstain from making a prediction.

Logistic Regression outputs probabilities, which gives you a notion that there is confidence in prediction. But in fact, it is not. $P(y|x)$ being 0.3, doesn't tell me that the model is confident in its prediction or not. All, we can say is that it believes 100% that class A is 0.3 and other class is 0.7.

Can we use confidence intervals of LR as some sort of confidence in prediction? E.g., larger the difference between upper and lower bound for a particular data point, less confident it is, and vice-versa?

If not, how can we build confidence in our prediction? Can anybody guide me to some paper or field of study?

  • 1
    $\begingroup$ You could use bootstrap samples to attain different values and calculate a CI based on these new outcomes. $\endgroup$ – peteR Feb 21 '19 at 7:41
  • $\begingroup$ yes you can use confidence interval calculations for logistic regression, and for more complicated models you have to use bootstrap samples. see stats.stackexchange.com/a/354660 $\endgroup$ – seanv507 Feb 21 '19 at 9:50

Your inference about $y$ is the Bernoulli distribution $$ \mathrm{prob}(y = y^\prime | x = x^\prime, \mathcal{D}, \mathcal{I}) = \begin{cases} 0.3 & y^\prime = 1 \\ 0.7 & y^\prime = 0 \end{cases} $$ The probability $0.3$ is itself the measure of your confidence in $y = 1$. So you should not confidently predict that $y = 1$, nor should you even confidently predict that $y = 0$.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.