2
$\begingroup$

I already referred this post post and post. Don't mark it as duplicate.

I am working on a binary classification problem using logistic regression. Loan default or not.

I have a requirement where I am told to predict the outcome classes for our unseen records and also report the confidence of predictions generated.

I am novice data scientist. So, am not really sure what is the difference between likelihood and confidence?

If I run logistic regression, I get likelihood measure like 70% probability of belonging to class 1 and 30% of probability of belonging to class 0.

My questions are as follows

a) Does likelihood and confidence mean the same? Is there any simple explanation that ordinary layman like me can understand.

b) Is there any tutorial that you can suggest which has on how to report confidence of predictions? When I use scikit-learn for logistic regression, I don't know how can I report confidence for the predictions?

c) Any idea on how can we generate the interval? In scikit-learn logistic regression tutorials that I find online, I only see probability/likelihood of an instance becoming label 1 or label 0. Can you guide me on how can we generate interval?

$\endgroup$
10
  • $\begingroup$ 1) You’re asking about the “likelihood” of an event, but that does not appear to be part of your requirements. Did your boss ask for a “likelihood” of an event? // 2) Consider asking your boss for clarification on what exactly is needed. // 3) Do you have an explicit requirement to give a confidence interval? I can think of a way to report the results without giving a confidence interval, yet I would answer the questions about predicted class membership and confidence about loan default. $\endgroup$
    – Dave
    Jan 18 at 1:30
  • 1
    $\begingroup$ 4) This seems like a perfect situation to predict tendency, not membership. Making a hard classification requires knowledge of what’s at stake. What do you lose when someone you predicted as safe winds up defaulting? What do you lose when someone safe is falsely regarded as risky and not approved for a loan? Those will influence how you use the probability output of your logistic regression. You don’t have to use the software default of $0.5$ as your decision threshold. $\endgroup$
    – Dave
    Jan 18 at 1:35
  • $\begingroup$ Hi @Dave - Yes, I wanted to compute/predict the likelihood and the confidence of my predictions. For ex: I can say that person A has a 90% likelihood of his loan being approved. But how confident am I about the inference/conclusion that we make from the model results. That's what they wish to know. Likelihood and confidence $\endgroup$
    – The Great
    Jan 18 at 2:28
  • $\begingroup$ 3) Yes, I do have an explicit requirement to report results with confidence (interval) etc. Can direct me to any resources that involves likelihood determination as well as confidence of the results? $\endgroup$
    – The Great
    Jan 18 at 2:30
  • 1
    $\begingroup$ It's typical to chat in the chat after you continue the discussion in chat. Let's migrate there. $\endgroup$
    – Dave
    Jan 18 at 22:17

1 Answer 1

3
$\begingroup$

My guess is that Bayesian (Logistic) Regression is what you're looking for.

In a regular Logistic Regression you find the (model) parameters $\beta$ that maximise the likelihood of your data (see Maximum Likelihood estimation). In a binary classification you use those parameters to calculate the probability $p$ of a given (test) sample $X_{test}$ of belonging to a positive class.

In Bayesian regression, on the other hand, you're not trying to find single best parameters $\beta$ that fit your data. Instead, you're trying to find the distribution of $\beta$s given your prior belief about the parameters (e.g. you can assume that they're coming from a normal distribution centered at $0$ with some standard deviation $\sigma$*) and using your data to update your prior belief, arriving at a posterior distribution of $\beta$s. You can use the means of those posterior distributions to do predictions - this should give you results similar (or equivalent**) to predictions done using estimates from a normal logistic regression. However, having an entire distribution of $\beta$s (as opposed to a having a single value) gives you an idea about the variability of those estimates. A posterior distribution with high variance indicates that your estimate of $\beta$ has lots of variability, i.e. given a slightly different training dataset, the MLE estimate of $\beta$ can change a lot. You can use the standard deviation of the posterior distribution to calculate the confidence interval for the distribution of $\beta$. Now instead of reporting $\beta = 0.1$ you can say that e.g. 95% confidence interval for $\beta$ is e.g. $[0.05, 0.15]$. I believe that this is what your requirements refer to as confidence.

Remark You can perform predictions using both $\beta$s corresponding to the lower and upper bound of the confidence interval and this will give you two distinct predictions).

Here is a Python example of Bayesian Logistic Regression performed using PyMC3 package.

* What consitutes a good prior is a topic for a separate debate. Gaussian distribution centered at 0 is a good default choice. It also has a regularization effect as it forces the parameters to be close to 0.

** Logistic Regression with uniform prior will give results equivalent to MLE estimates.

$\endgroup$
10
  • 1
    $\begingroup$ thanks for the help. upvoted $\endgroup$
    – The Great
    Jan 18 at 10:09
  • $\begingroup$ the machine always predicts “yes” with a probability between 0 and 1. So, is that called as our confidence score? $\endgroup$
    – The Great
    Jan 18 at 10:13
  • 1
    $\begingroup$ The confidence (interval) in this case would refer to $\beta$s, not the predictions themselves (at least not directly). Regular logistic regression gives you only one (MLE) estimate of $\beta$. Using this $\beta$ you can make a prediction, e.g. say that for $X_{test}$ it says that the probability of class 1 is 60%. In Bayesian Logistic Regression you get a range of $\beta$s (corresponding to the 95% confidence interval). If you use those two extreme $\beta$s to do predictions on the same test sample you'll get a range of probabilities of belonging to the class 1 (e.g. 55% - 65%). $\endgroup$ Jan 18 at 10:36
  • 2
    $\begingroup$ @TheGreat It is on you to define what confidence means. From what you wrote in your comments to me, it seems like you are interested in model validation to show a colloquial “confidence” that the model is a good one, rather than formal confidence intervals. $\endgroup$
    – Dave
    Jan 18 at 10:44
  • 1
    $\begingroup$ @TheGreat I don't know, although that would be my guess. Or it could be simply the difference between the $P(X = 1)$ and $P(X = 0)$, i.e. how confident your model predictions are for a given sample (i.e. 51%/49% is not confident, 99%/1% is confident). $\endgroup$ Jan 18 at 10:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.