4
$\begingroup$

I think I already have the answer, however, I wish for some confirmation that I am not missing anything here. This sort of asks the same thing, but I want to double-check.


Logistic regression can be motivated via generalized linear models.

GLM, in essence, says that we model the transformed (“linked” so to speak) expected value $\mu$ of a variable $Y$ given covariates/features as a linear function. Let's call the link function $g()$. In case of the classical linear regression model this function would simply be the identity function. If $Y$ is binary, the expected value is equal to $p = P(Y = 1)$. In the logistic regression model, we model the log-odds as a linear function:

$$ \log\left(\frac{p}{1-p}\right) = \beta_0 + \beta_1x_1 + \dots + \beta_Kx_K$$

So the assumption is that the log-odds are adequately described by a linear function. The logit function, however, clearly is not a linear function. Yet, it is reasonably approximated by a linear function if we truncate the probability range to something like $0.05 < p < 0.95$.

Question: why do we model the log-odds as a linear function when it is nonlinear for small and large probabilities?

My answer would be that since we are interested in the expected value, we assume (!) that the relevant range of probabilities we are trying to estimate does not contain these “extreme” probabilities. Hence, in essence, we simply ignore the nonlinearity.

Correct?

$\endgroup$
4
  • 2
    $\begingroup$ Because it is easiest to assume linearity, at least as a starting point. It is the same reason we assume linearity in ordinary regression, even though the assumption is almost always false (in both regressions) to one degree or another. $\endgroup$ Jan 3, 2021 at 22:08
  • 2
    $\begingroup$ Adding to the comment by @BigBendRegion, you seem be confusing two things: (1) logit being nonlinear in "p" (2) assuming that the logit of $p$ is linear in the covariates. The first point has no bearing on the second point unless somehow you believe that the probabilities themselves should be linearly dependent on the covarites (which is perhaps even more absurd considering that $p$ has to remain in $[0,1]$). $\endgroup$
    – passerby51
    Jan 3, 2021 at 22:46
  • 1
    $\begingroup$ Glad I asked! I was indeed confusing those two things! The log odds are assumed to be linearly related to the covariates (which we could relax by adding quadratic or cubic terms etc) not to the probabilities (p). @passerby51 if you want to write an answer I’d accept it. If not I’ll write one myself in a couple of days. Thanks $\endgroup$
    – Manuel R
    Jan 4, 2021 at 8:51
  • $\begingroup$ @ManuelR, made the comment into an answer and expanded a bit. $\endgroup$
    – passerby51
    Jan 4, 2021 at 17:57

1 Answer 1

2
$\begingroup$

A comment turned into an answer:

You seem be confusing two things: (1) The "logit" being nonlinear in $p$ (2) assuming that the logit of p is linear in the covariates. The first point has no bearing on the second point unless somehow you believe that the probabilities themselves should be linearly dependent on the covariates, which is perhaps even more absurd considering that p has to remain in [0,1].

  • The best way to see why logistic regression makes sense is to try to model the probability $p$ as a function of $x = (x_1\dots,x_{K})$. You quickly realize that perhaps you need some sort of transformation that restricts the values to $[0,1]$ and some thought might lead to a model like $$ p = \phi(\beta^T x) $$ where $\phi(\cdot)$ is a function from $\mathbb R$ to $[0,1]$. One example will be $\phi = \text{logit}^{-1}$ which leads to logistic regression. Another example is $\phi = $ CDF of the standard normal distribution which leads to Probit regression, and so on.

  • You can always make the model more complex by say assuming $p = \phi( P_\beta(x))$ where $P_\beta(x)$ is a polynomial in $x$ of degree higher than 1.

  • The logit case also has the following interpretation: Let the binary observation be $Y$ with density (i.e., PMF) $p(y) = p^{y} (1-p)^{1-y}$ for $y \in \{0,1\}$. This is an exponential family $$ p(y) = \exp( y \theta - \log(1 +e^{\theta})) $$ with canonical/natural parameter $\theta = \log\frac{p}{1-p}$. The logistic regression assumes this canonical parameter to be linear in the covariates.

  • A similar consideration as point 1 above goes into modeling a parameter which takes values in $[0,\infty)$ such as a rate $\lambda$. Then, again, a natural first model is $\lambda = \phi(\beta^T x)$ where $\phi(\cdot)$ maps $\mathbb R$ to $[0,\infty)$ and a natural choice for $\phi$ is $\phi(x) = e^x$.

$\endgroup$
1
  • 2
    $\begingroup$ damn. way over my head. $\endgroup$ Jul 2, 2021 at 8:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.