8
$\begingroup$

What I've seen very often, is that people are saying, that logistic and probit regression are giving very similar results, however, logistic regression is more interpretabe. And in this question, I want to understand why.

In the logistic regression, interpretability lies on the log of the odds:

$$\log\left(\frac{p}{1 - p}\right) = \beta_0 + \sum_{i = 1}^n \beta_iX_i,$$

because if we take the derivative with respect to $j-$ th variable, we exactly obtain the $j-$th coefficient:

$$\frac{\partial\log\left(\frac{p}{1 - p}\right)}{\partial X_j} = \beta_j.$$

Because this happens, people are saying, that increase $X_j$ of one unit, increase the odds (i.e. $\frac{p}{1-p}$) of $\exp(\beta_j)$ units.

In the probit regression we can come up with the exactly same thing. We have the following relationship:

$$p = \Phi\left(\beta_0 + \sum_{i = 1}^n \beta_i X_i\right),$$

where $\Phi$ is the PDF of standard normal distribution. From this equation we have that:

$$Q(p) = \beta_0 + \sum_{i = 1}^n \beta_i X_i,$$

where $Q$ is a quantile function of standard normal distribution.

Now if we take the derivative :

$$\frac{\partial Q(p)}{\partial X_j} = \beta_j$$

So can we say the same? What I would like to say, is that increase $X_j$ by one unit, increases quantile function by one unit, and therefore, increase $X_j$ by one unit, increases $p$ by $\Phi(\beta_j)$ units. If that's true, why people are saying, that probit regression is less interpretable?

$\endgroup$
8
  • 4
    $\begingroup$ Perhaps because the ratio of two odds --- the "odd ratio" --- is a concept well known beyond logistic regression? I've never seen anyone using the ratio of two quantiles, much less heard a name for it. $\endgroup$
    – Igor F.
    Sep 7, 2022 at 11:21
  • 3
    $\begingroup$ Having to take a derivative to make something interpretable is not what I'd recommend. $\endgroup$ Sep 7, 2022 at 11:55
  • 2
    $\begingroup$ "increases $p$ by $\Phi(β_j)$ units" looks peculiar - if $\beta_j=1$, are you saying an increase of $X_j$ by one unit increases $p$ by $0.841$? You would not want to do that twice. Or are you saying it increases $p$ by $\Phi(\hat Y +\beta_j)-\Phi(\hat Y)$ perhaps approximately $\beta_j \phi(\hat Y)$ - neither of which seem easy to understand $\endgroup$
    – Henry
    Sep 7, 2022 at 11:58
  • 3
    $\begingroup$ Count me as one of them, in terms of coefficient interpretation. $\endgroup$ Sep 7, 2022 at 13:06
  • 2
    $\begingroup$ But the function $\Phi(x)$ is very interpretable not? A change of $x$ is a change of one standard variation on a normal bell curve. It is maybe not as easy to compute the exact probability as with log-odds, but it is not difficult to interpret. There are even fields where they describe probabilities in terms of the number of $\sigma$. $\endgroup$ Sep 7, 2022 at 13:09

2 Answers 2

14
$\begingroup$

Logistic regression is a model for probabilities of binary events. Another concept that is closely related to probabilities is odds, i.e. ratios of probabilities. If the probability of observing a binary event is $p$, the odds of observing it is $\tfrac{p}{1-p}$. It's a fairly simple and commonly understood concept. Logistic regression predicts log-odds. They are "simpler" to interpret because odds are already related to probabilities of binary events, while normal quantiles do not directly translate to them in a meaningful way. If the predicted quantile is $Q(p)$, how "likely" is this to happen? To answer the question, you need to translate the value to probability, while in the case of odds it just re-phrases the question to probability relative to the probability of the opposite event.

$\endgroup$
5
$\begingroup$

Here's another way to think about this:

Let $\theta = β_0 +\sum_{i=1}^n \beta_i X_i $ be your linear predictor. If $X_j$ increases by one unit, the linear predictor increases by $\beta_j$. What can we say about the corresponding change in probability, $|p(\theta+ \beta_j)-p(\theta)|$?

In a logistic regression model, the probability is given by $p(\theta)=e^\theta/(1+e^\theta)$, the inverse of the logit (log-odds) function. It is straightforward to show that the derivative of this function is maximised at $\theta=0$, and that $p'(0)=1/4$. We deduce that $|p(\theta+ \beta_j)-p(\theta)| < \frac{|\beta_j|}{4}$. In other words, the maximum effect a unit change in the $j$th variable can have on the probability is equal to its coefficient divided by 4.

Can you work out the equivalent for the probit model?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.