Why are log odds modelled as a linear function?

Question

I think I already have the answer, however, I wish for some confirmation that I am not missing anything here. This sort of asks the same thing, but I want to double-check.

---

Logistic regression can be motivated via generalized linear models.

GLM, in essence, says that we model the transformed (“linked” so to speak) expected value $\mu$ of a variable $Y$ given covariates/features as a linear function. Let's call the link function $g()$. In case of the classical linear regression model this function would simply be the identity function. If $Y$ is binary, the expected value is equal to $p = P(Y = 1)$. In the logistic regression model, we model the log-odds as a linear function:

$$ \log\left(\frac{p}{1-p}\right) = \beta_0 + \beta_1x_1 + \dots + \beta_Kx_K$$

So the assumption is that the log-odds are adequately described by a linear function. The logit function, however, clearly is not a linear function. Yet, it is reasonably approximated by a linear function if we truncate the probability range to something like $0.05 < p < 0.95$.

Question: why do we model the log-odds as a linear function when it is nonlinear for small and large probabilities?

My answer would be that since we are interested in the expected value, we assume (!) that the relevant range of probabilities we are trying to estimate does not contain these “extreme” probabilities. Hence, in essence, we simply ignore the nonlinearity.

Correct?

Answer 1

A comment turned into an answer:

You seem be confusing two things: (1) The "logit" being nonlinear in $p$ (2) assuming that the logit of p is linear in the covariates. The first point has no bearing on the second point unless somehow you believe that the probabilities themselves should be linearly dependent on the covariates, which is perhaps even more absurd considering that p has to remain in [0,1].

• The best way to see why logistic regression makes sense is to try to model the probability $p$ as a function of $x = (x_1\dots,x_{K})$. You quickly realize that perhaps you need some sort of transformation that restricts the values to $[0,1]$ and some thought might lead to a model like $$ p = \phi(\beta^T x) $$ where $\phi(\cdot)$ is a function from $\mathbb R$ to $[0,1]$. One example will be $\phi = \text{logit}^{-1}$ which leads to logistic regression. Another example is $\phi = $ CDF of the standard normal distribution which leads to Probit regression, and so on.

• You can always make the model more complex by say assuming $p = \phi( P_\beta(x))$ where $P_\beta(x)$ is a polynomial in $x$ of degree higher than 1.

• The logit case also has the following interpretation: Let the binary observation be $Y$ with density (i.e., PMF) $p(y) = p^{y} (1-p)^{1-y}$ for $y \in \{0,1\}$. This is an exponential family $$ p(y) = \exp( y \theta - \log(1 +e^{\theta})) $$ with canonical/natural parameter $\theta = \log\frac{p}{1-p}$. The logistic regression assumes this canonical parameter to be linear in the covariates.

• A similar consideration as point 1 above goes into modeling a parameter which takes values in $[0,\infty)$ such as a rate $\lambda$. Then, again, a natural first model is $\lambda = \phi(\beta^T x)$ where $\phi(\cdot)$ maps $\mathbb R$ to $[0,\infty)$ and a natural choice for $\phi$ is $\phi(x) = e^x$.