9
$\begingroup$

Within logistic regression framework, odds are defined as:

$$\frac{p(X)}{1-p(X)}=e^{\beta_0+\beta_{1}X}. $$

In the textbook "An introduction to Statistical Learning" (Gareth James, et al., 2013), on page 132 I read:

In contrast, in a logistic regression model, increasing $X$ by one unit changes the log odds by $\beta_1 (4.4)$, or equivalently it multiplies the odds by $e^{\beta_1} (4.3)$.

How come one unit change in $X$ multiplies the odds by $e^{\beta_1}$?

My reasoning is the following:

$$\frac{\partial e^{\beta_0+\beta_{1}X}}{\partial X}= \beta_1e^{\beta_0+\beta_{1}X},$$

therefore a unit change in $X$ should change $\frac{p(X)}{1-p(X)}$ by $\beta_1e^{\beta_0+\beta_{1}X}$. Isn't this correct?

EDIT In citation, we have (4.3) and (4.4) equations, which are respectively given

$$\frac{p(X)}{1-p(X)}=e^{\beta_0+\beta_{1}X}. $$

$$\log\left(\frac{p(X)}{1-p(X)}\right)=\beta_0+\beta_{1}X. $$

$\endgroup$
1
  • $\begingroup$ $\log\left(\frac{p(x)}{1-p(x)}\right)=\beta_0+\beta_{1}x$ is a linear model for the log-odds and has a derivative of $\beta_1$ and is the only place the derivative is meaningful. Increasing $x$ to $x+1$ adds $\beta_1$ to the log-odds $\big($i.e. multiplies the odds by $e^{\beta_1}\big)$ while increasing $x$ to $x+\delta$ adds $\beta_1\delta$ to the log-odds $\big($i.e. multiplies the odds by $e^{\beta_1 \delta}\big)$. $\endgroup$
    – Henry
    Commented May 28 at 10:56

4 Answers 4

6
$\begingroup$

Suppose you have a function in $x$, say $f(x)=x^2$. The derivative is $f'(x)=2x$. This means that if $x$ increases 1 point, the function $x^2$ would increase by $2x$, if we would "believe" the derivative, or to put it differently: if we would assume a linear increase of the function equal to the derivative. However, such linear increase is not correct, as can easily be verified. For example, if $x=1$ then $x^2=1$ and if $x=2$, $x^2=4$ so, if $x$ increases from 1 to 2, $x^2$ increases 3 points. However, if we "believe" the derivative taken at $x=1$ then the increase would be $2x=2$, so this would mean that at $x=2$ the function $x^2$ would have value $1+2=3$, which is clearly wrong!

Now to your odds question. Your derivative is right. So if $x$ increases by 1 point, the odds would increase by $b_1e^{b_0+b_1x}$, if we were to believe the derivative. So, an increase of 1 point in $x$ would lead to a new odds value being $e^{b_0+b_1x} + b_1e^{b_0+b_1x}$. But note that this increase has to be understood additive, not multiplicative! To arrive at the multiplicative increase (again based on the derivative), we would have to divide the new "derivative based" odds by the "old" odds, leading to: $\frac{e^{b_0+b_1x} + b_1e^{b_0+b_1x}}{e^{b_0+b_1x}}=1+b_1$. But clearly this "derivative based" increase is not equal the (multiplicative) increase of the odds that we all know, namely $e^{b_1}$.

Indeed, as was already mentioned in a comment, using the (linear) derivative of a nonlinear function to approximate the true increase in that nonlinear function, will in general lead to the wrong increase.

Forgot to mention that I always explain the $e^{b_1}$ multiplication of the odds quite simply, without referring to derivatives. If $x$ increases 1 point, then the odds goes from $e^{b_0+b_1x}$ to $e^{b_0+b_1(x+1)}=e^{b_0+b_1x+b_1}=e^{b_0+b_1x}e^{b_1}$ or: the odds is multiplied by $e^{b_1}$.

$\endgroup$
3
  • $\begingroup$ I see, thank you! So in other words, the difference between the results, i.e., $\beta_{1} e^{\beta_0 + \beta_{1} X}$ vs $e^{\beta_1}e^{\beta_0 + \beta_1 X}$ comes from linear approximation of changes in (non-linear) odds function through derivatives, right? $\endgroup$
    – Sane
    Commented May 27 at 15:10
  • 1
    $\begingroup$ Correct. The derivative gives the instantaneous change at a given x value, i.e. for an infinitisemally small change in x, and for a change of 1 point. $\endgroup$
    – BenP
    Commented May 27 at 15:21
  • $\begingroup$ $Sane The difference also comes from Additive vs. Multiplicative change. $\endgroup$
    – BenP
    Commented May 27 at 16:21
6
$\begingroup$

Notice your citation refers to the log-odds, i.e., $$ \log\left(\frac{p(X)}{1-p(X)}\right)=\log(e^{\beta_0+\beta_{1}X})=\beta_0+\beta_{1}X $$ Here, we do have $$ \frac{\partial (\beta_0+\beta_{1}X)}{\partial X}= \beta_1,$$ which, upon exponentiating for the odds rather than log-odds scale, gives $e^{\beta_1}$.

$\endgroup$
11
  • $\begingroup$ And note that derivatives (elasticities) don’t necessarily have anything to do with the problem being solved. As stated it is about estimating changes in something over a nonzero $\Delta$. $\endgroup$ Commented May 27 at 11:35
  • $\begingroup$ Thank you for your response. My citation refers to (4.3) equation which is odds equation. Why I don't get the same result from $\frac{\partial e^{\beta_0+\beta_1 X}}{\partial X}$? As we know $e^{\beta_0+\beta_1 X}$ is odds equation, therefore derivative should show change in odds, right? $\endgroup$
    – Sane
    Commented May 27 at 12:28
  • $\begingroup$ The citation says "log odds", no? As to why you do not get the same result - a consequence of the fact that a nonlinear function of a derivative is not the same thing as a derivative of a nonlinear function, if I understand your question correctly? $\endgroup$ Commented May 27 at 12:36
  • $\begingroup$ I understand from the textbook that "odds". I edited citation in my question, and added an Edit with clear definition of corresponding equations. $\endgroup$
    – Sane
    Commented May 27 at 13:20
  • 1
    $\begingroup$ Read the OP again!! Instantaneous change not mentioned and one-unit change explicitly mentioned! $\endgroup$ Commented May 27 at 19:10
6
$\begingroup$

Using equations (4.3) and (4.4):

$$\frac{p(X+1)}{1-p(X+1)} = e^{\beta_0+\beta_{1}(X+1)} = e^{\beta_{1}*1} * e^{\beta_0+\beta_{1}X } = e^{\beta_{1}} * \frac{p(X)}{1-p(X)} $$

$$log(\frac{p(X+1)}{1-p(X+1)})=\beta_0+\beta_{1}(X+1) = \beta_0+\beta_{1}X + \beta_{1}*1 = log(\frac{p(X)}{1-p(X)}) + \beta_{1} $$

$\endgroup$
4
  • $\begingroup$ Thank you. I see your point -- but how would one get your results by taking derivative of the functions? If we take derivative of $e^{\beta_0+\beta_{1}X}$, we get $\beta_{1} e^{\beta_0+\beta_{1}X}$. This means that any small change in $X$, multiplies the odds by $\beta_1$, not by $e^{\beta_{1}}$, correct? $\endgroup$
    – Sane
    Commented May 27 at 14:23
  • 1
    $\begingroup$ You are fixated on derivatives. Please read what others have written above. To compute a change in something you subtract one thing from the other (additive change) or divide one by the other (fold change; ratio), and for both you specify the starting and endpoint points. For a linear predictor effect as you have, a change in log-odds or fold change in odds doesn't depend on the starting point; it only depends on the difference in the two predictor values. $\endgroup$ Commented May 27 at 19:23
  • $\begingroup$ You still can get the same results through derivatives for a linear function -- $y = \beta_0 + \beta_{1 }x$ and $y' = \beta_{0} + \beta_{1} (x+1)$, therefore $y'-y = \beta_{1}$. Alternatively, $\frac{\partial y}{\partial x} = \beta_{1}$. The results coincide due to the linear function. Regarding the odds equation, the results do not due to non-linear function. And i was fixating on derivatives as i wanted to understand where the logic is breaking. $\endgroup$
    – Sane
    Commented May 28 at 6:43
  • 2
    $\begingroup$ This fixation is hard to understand. The way regression models are interpreted in general is by studying how $\hat{Y}$ changes when an $X$ changes, holding all other $X$ constant. This has absolutely nothing to do with instantaneous changes but rather fixes changes in $X$ at nonzero values. For logistic models we look at both changes in logits and changes in probabilities. The default I use in the R rms package gives you interquartile-range $X$ effects on log odds and odds ratios, and partial effect plots gives you the general result which is especially needed when $X$ is nonlinear. $\endgroup$ Commented May 28 at 11:41
1
$\begingroup$

My reasoning is the following:

$$\frac{\partial e^{\beta_0+\beta_{1}X}}{\partial X}= \beta_1e^{\beta_0+\beta_{1}X},$$

Your reasoning relates to a linear approximation which describes the change for a tiny little step, and not for an entire unit change.

You can compare this to the wheat and chessboard problem. The rate of change increases.

If the changes are small then your increase is a good approximation since

$$\frac{\text{new value}}{\text{old value}} = \frac{e^{\beta_0+\beta_{1}x} + 1\cdot \beta_1e^{\beta_0+\beta_{1}x}}{e^{\beta_0+\beta_{1}x}} = 1+\beta_1 \approx e^{\beta_1}$$

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.