Problem understanding the logistic regression link function

Question

I am trying to learn the logistic regression model. I came to know that there is no linear relationship between predictor variables and response variables since response variables are binary (dichotomous). The link function used for logistic regression is logit which is given by $$ \log \frac {p}{1 - p} = \beta X $$ This tells that the log odds is a linear function of input features. Can anyone give me the mathematical interpretation of how the above relation becomes linear i.e. how logistic regression assumes that the log odds are linear function of input features? Since I am poor at statistics, I can't understand complex mathematical answer.

Answer 1

We can rationalize this as follows:
Underlying logistic regression is a latent (unobservable) linear regression model:

$$y^* = X\beta + u$$

where $y^*$ is a continuous unobservable variable (and $X$ is the regressor matrix). The error term is assumed, conditional on the regressors, to follow the logistic distribution, $u\mid X\sim \Lambda(0, \frac {\pi^2}{3})$.

We assume that what we observe, i.e. the binary variable $y$, is an Indicator function of the unobservable $y^*$:

$$ y = 1 \;\;\text{if} \;\;y^*>0,\qquad y = 0 \;\;\text{if}\;\; y^*\le 0$$

Then we ask "what is the probability that $y$ will take the value $1$ given the regressors (i.e. we are looking at a conditional probability). This is

$$P(y =1\mid X ) = P(y^*>0\mid X) = P(X\beta + u>0\mid X) = P(u> - X\beta\mid X) \\= 1- \Lambda (-Χ\beta) = \Lambda (X\beta) $$

the last equality due to the symmetry property of the logistic cumulative distribution function.

So we have obtained the basic logistic regression model

$$p=P(y =1 \mid X) = \Lambda (X\beta) = \frac 1 {1+e^{-X\beta}}$$

After that, the other answers give you how we manipulate this expression algebraically to arrive at $$\log \frac {p}{1 - p} = X\beta $$

It is therefore the initial linear assumption/specification related to the Latent variable $y^*$, that leads to this last relation to hold.

Note that $\log \frac {p}{1 - p}$ is not equal to the latent variable $y^*$ but rather $y^* = \log \frac {p}{1 - p} + u$

Answer 2

First of all let me re-write your logit model as $$g(x)=\ln \frac{p(x)}{1-p(x)}=\beta X,$$ just to emphasize that those $p(x)$ depends on your explanatory random variables, $X$. When we say that $g(x)$ is "linear" it means that this relation is linear with respect to the parameters $\beta$'s and not in those $X$'s. For example let's say you have two explanatory random variables, $X_1$ and $X_2$. Then not only $$g(x)=\ln \frac{p(x)}{1-p(x)}=\beta_0+\beta_1X_1+\beta_2X_2,$$ but also $$g(x)=\ln \frac{p(x)}{1-p(x)}=\beta_0+\beta_1X_1^2+\beta_2X_2^3,$$ and $$g(x)=\ln \frac{p(x)}{1-p(x)}=\beta_0+\beta_1\sin{(X_1)}+\beta_2\sin{(X_2)},$$ are all linear with respect to $\beta_i$'s for $i=0,1,2$. However, the model $$g(x)=\ln \frac{p(x)}{1-p(x)}=\beta_0+\beta_0\beta_1X,$$ is not linear in $\beta_i$'s because of that product $\beta_0\beta_1$.

Answer 3

For logistic Regression, our hypothesis is: $$ h_\theta (X) = p = \frac{1}{1 - \exp(-\beta.X)} $$ now simplifying it: $$ 1 - \exp(-\beta.X) = \frac{1}{p} $$ $$ 1 - \frac{1}{p} = \exp(-\beta.X)$$ taking log on both sides, and simplifying it, $$ \ln{\frac{p}{1-p}} = \beta.X $$

now, if we look at this final equation, the LHS of it is logit fuction of $p$, and RHS is the dot product of two vectors, its expansion will look like:

$$ \beta.X = \beta_0 + \beta_1.X_1 + \beta_2.X_2 + ... $$ parameters $ \beta_{n * 1} $ and input features $ X_{1 * n} $. Since, the parameters $ \beta $ are learned due to our learning algorithms and are constant, and there is involvement of input features in first degree only (power 1) e.g. terms like $X_1^2, X_1.X_2^2$ do not appear, and no mixed features terms like $X_1.X_2$, the logit of our hypothesis/or probability is linear function/interpretation of input features $ X$.