It is well known that for a linear regression problem $y=X\beta+\epsilon$, we have a closed form solution $\hat{\beta}=(X^TX)^{-1}X^Ty$.

Then, it's natural to ask the similar question about logistic regression, which has also been discussed (e.g. here).

However, I don't understand it well that what the reason there to forbid us from just inverse the logistic transformation and use the closed form solution for linear regression again.

For example, for a logistic regression problem $<y, X>$, since we model it in such a way: $$ y = \dfrac{1}{1+e^{-X\beta}} $$ why can't we inverse that function to get $X\beta=\log\dfrac{y}{1-y}=y'$, and simply following the closed form to get $\hat{\beta}=(X^TX)^{-1}X^Ty'$?

I hope the reason is more than "$1-y$ will be zero for $y=1$" because I believe we can always figure out a way to work around these computation issues.


Thanks for @Chaconne's suggestion of this almost identical question.

  • 5
    $\begingroup$ Does this question help? stats.stackexchange.com/questions/122103/… Basically a GLM specifies a model for a transformation of the mean and a distribution for the responses, rather than directly modeling the responses $\endgroup$
    – jld
    Sep 20 '17 at 15:30
  • $\begingroup$ Thanks @Chaconne This is truly helpful. I edited the question with a follow-up. $\endgroup$ Sep 20 '17 at 18:57
  • $\begingroup$ related question here. and do not know if AdamO's answer to my question help. $\endgroup$
    – Haitao Du
    Sep 20 '17 at 19:03

Browse other questions tagged or ask your own question.