I am trying to reproduce the following example of logistic regression with a transformed linear regression:

am.glm <- glm(am ~ hp + wt, data=mtcars, family=binomial)
newdata <- data.frame(hp=120, wt=2.8)
predict(am.glm, newdata, type="response") 
##         1 
## 0.6418125

The equation for the probability of $Y=1$ is the following: $$ P(Y=1) = {1 \over 1+e^{-(b_0+\sum{(b_iX_i)})}} $$

So I tried something like this:

am.lm <- lm(am ~ 1/(1+exp(-(hp + wt))),data=mtcars)
predict(am.lm, newdata)
##       1 
## 0.40625

So this is obviously wrong! (I also tried transforming the given value but nothing worked so far).

My question
How would I have to set up logistic regression with explicitly specifying the formula for the non-linear transformation of the linear model?


The short answer is "you don't". They don't correspond.

Logistic regression is not a transformed linear regression.

Even though $E(Y)$ ($=P(Y=1)$) may be written as $\text{logit}(X\beta)$, and so seemingly linearized, you can't transform the $y$ values to make a linear regression, nor can you fit a nonlinear least squares model (transforming the x's) that reproduces the logistic regression. [You may be able to fit a function of the logistic form via nonlinear least squares but it won't have the same estimates.] Answers to this question gives some additional details.

The observations enter the model estimation quite differently.

Logistic regression is fitted by maximum likelihood estimation for a binomial model with the natural link function (which is the logit for the binomial). That is, the data are seen as $y_i\sim\text{binomial}(n_i,1/(1+e^{-X_i\beta}))$

(Where $X_i\beta=\beta_0+\sum_j \beta_j x_{ij}$ is the linear predictor for casec$i$.)

Once you find the MLE for the model, you could find a set of weights and a set of pseudo-observations such that a linear model yields the parameter estimates, but the connection to the original data is quite indirect and in general you can't get to the point of doing so until you've already found the solution.

It's not particularly useful to think of that as a transformation of the data.

  • $\begingroup$ Thank you. So the main problem of transforming logistic regression to linear regression is not the non-linear transformation but the estimation part (least squares vs maximum likelihood for a binomial model)? $\endgroup$
    – vonjd
    Jul 20 '15 at 11:41
  • 1
    $\begingroup$ In a sense, yes; the observations enter the model estimation quite differently; I should say that in my answer. $\endgroup$
    – Glen_b
    Jul 20 '15 at 12:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.