I am trying to learn the logistic regression model. I came to know that there is no linear relationship between predictor variables and response variables since response variables are binary (dichotomous). The link function used for logistic regression is logit which is given by $$ \log \frac {p}{1 - p} = \beta X $$ This tells that the log odds is a linear function of input features. Can anyone give me the mathematical interpretation of how the above relation becomes linear i.e. how logistic regression assumes that the log odds are linear function of input features? Since I am poor at statistics, I can't understand complex mathematical answer.

up vote 4 down vote accepted

We can rationalize this as follows:
Underlying logistic regression is a latent (unobservable) linear regression model:

$$y^* = X\beta + u$$

where $y^*$ is a continuous unobservable variable (and $X$ is the regressor matrix). The error term is assumed, conditional on the regressors, to follow the logistic distribution, $u\mid X\sim \Lambda(0, \frac {\pi^2}{3})$.

We assume that what we observe, i.e. the binary variable $y$, is an Indicator function of the unobservable $y^*$:

$$ y = 1 \;\;\text{if} \;\;y^*>0,\qquad y = 0 \;\;\text{if}\;\; y^*\le 0$$

Then we ask "what is the probability that $y$ will take the value $1$ given the regressors (i.e. we are looking at a conditional probability). This is

$$P(y =1\mid X ) = P(y^*>0\mid X) = P(X\beta + u>0\mid X) = P(u> - X\beta\mid X) \\= 1- \Lambda (-Χ\beta) = \Lambda (X\beta) $$

the last equality due to the symmetry property of the logistic cumulative distribution function.

So we have obtained the basic logistic regression model

$$p=P(y =1 \mid X) = \Lambda (X\beta) = \frac 1 {1+e^{-X\beta}}$$

After that, the other answers give you how we manipulate this expression algebraically to arrive at $$\log \frac {p}{1 - p} = X\beta $$

It is therefore the initial linear assumption/specification related to the Latent variable $y^*$, that leads to this last relation to hold.

Note that $\log \frac {p}{1 - p}$ is not equal to the latent variable $y^*$ but rather $y^* = \log \frac {p}{1 - p} + u$

  • thank you so much. That was the answer I was looking for. – Bibek Subedi Dec 30 '13 at 3:50
  • Note that while assuming logistic errors for the latent response variable leads to a logistic link, assuming Gaussian errors leads to a probit link. Often enough a logistic link is just assumed with no appeal to latent variables; after all in empirical modelling you can add higher-order terms to your heart's content. The usual choice of a logistic link by default is because it's the canonical link. – Scortchi Jan 9 '14 at 22:58
  • @Scortchi Indeed. But the latent-variable interpretation provides intuition. Also, it offers a rationale for choosing a symmetric around zero distribution for the error term (given the constant term), (like the logistic or the normal, or even others). – Alecos Papadopoulos Jan 9 '14 at 23:48
  • 1
    @statBeginner It is not a step, just a re-writing of the same thing. Early on (see beginning of answer above) we have assumed that $u$ follows a logistic distribution. $\Lambda$ is the cumulative distribution function of the logistic distribution. The cumulative distribution function of a continuous random variable expresses "smaller than" (which would equal $\Lambda()$)or "greater than" ($=1-\Lambda()$) probabilities. – Alecos Papadopoulos Oct 29 '14 at 20:27
  • 1
    @statBeginner Yes. And if we assume that the error follows a normal, we get the "Probit" model, while if we assume that the error follows a uniform, we get the "Linear Probability" model. – Alecos Papadopoulos Oct 29 '14 at 20:35

First of all let me re-write your logit model as $$g(x)=\ln \frac{p(x)}{1-p(x)}=\beta X,$$ just to emphasize that those $p(x)$ depends on your explanatory random variables, $X$. When we say that $g(x)$ is "linear" it means that this relation is linear with respect to the parameters $\beta$'s and not in those $X$'s. For example let's say you have two explanatory random variables, $X_1$ and $X_2$. Then not only $$g(x)=\ln \frac{p(x)}{1-p(x)}=\beta_0+\beta_1X_1+\beta_2X_2,$$ but also $$g(x)=\ln \frac{p(x)}{1-p(x)}=\beta_0+\beta_1X_1^2+\beta_2X_2^3,$$ and $$g(x)=\ln \frac{p(x)}{1-p(x)}=\beta_0+\beta_1\sin{(X_1)}+\beta_2\sin{(X_2)},$$ are all linear with respect to $\beta_i$'s for $i=0,1,2$. However, the model $$g(x)=\ln \frac{p(x)}{1-p(x)}=\beta_0+\beta_0\beta_1X,$$ is not linear in $\beta_i$'s because of that product $\beta_0\beta_1$.

  • (+1) Clear explanation. Although I was asking something little different, your answer covers above 70% of my question. – Bibek Subedi Dec 27 '13 at 8:36

For logistic Regression, our hypothesis is: $$ h_\theta (X) = p = \frac{1}{1 - \exp(-\beta.X)} $$ now simplifying it: $$ 1 - \exp(-\beta.X) = \frac{1}{p} $$ $$ 1 - \frac{1}{p} = \exp(-\beta.X)$$ taking log on both sides, and simplifying it, $$ \ln{\frac{p}{1-p}} = \beta.X $$

now, if we look at this final equation, the LHS of it is logit fuction of $p$, and RHS is the dot product of two vectors, its expansion will look like:

$$ \beta.X = \beta_0 + \beta_1.X_1 + \beta_2.X_2 + ... $$ parameters $ \beta_{n * 1} $ and input features $ X_{1 * n} $. Since, the parameters $ \beta $ are learned due to our learning algorithms and are constant, and there is involvement of input features in first degree only (power 1) e.g. terms like $X_1^2, X_1.X_2^2$ do not appear, and no mixed features terms like $X_1.X_2$, the logit of our hypothesis/or probability is linear function/interpretation of input features $ X$.

  • @ bistaumanga: linearity is understood with respect to the parameters i.e. $\beta'$s and not the $X$'s. Even if you have terms like $X_1^2$ or $X_1.X_2^2$ you can simply let $Z_1=X_1^2$ and $Z_2=X_1.X_2^2$ and your model remains linear i.e. $$\ln\frac{p}{1-p}=\beta_0+\beta_1Z_1+\beta_2Z_2.$$ – Stat Dec 27 '13 at 9:04
  • exactly, in that case we would treat $Z_1$, $Z_2$ as input features.....whatever the feature, the model does it linearly, even though we do some preprocessing before running the model to the features... Isn't it like that? – bistaumanga Dec 27 '13 at 10:14

Your Answer

 
discard

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.