2
$\begingroup$

I don't understand something very fundamental about Logistic Regression.

For Linear Regression and for Naive Bayes there is a closed form for calculating parameters.

For some reason, logit regression doesn't have such a form. Do you know why?

$\endgroup$
  • $\begingroup$ I edited your "LR" to logit regression. Although it is evident what you mean, "LR" could also mean Linear Regression, especially given your own capitalisation. $\endgroup$ – Nick Cox Mar 5 '15 at 9:37
4
$\begingroup$

In the case of OLS, $f(\beta)=\frac12\|y - X\beta\|_2^2$, the gradient is $\nabla f(\beta)=X^T(X\beta - y)$. The first order condition is $\nabla f(\beta)=0$. Thus, at the optimum, $X^T(X\beta - y)=X^TX\beta - X^Ty = 0$. Solve for $\beta$ as $\beta=(X^TX)^{-1}X^Ty$. Thus, we are able to analytically solve for the optimum.

This is not the case for logistic regression, i.e. there is no analytic solution for the optimisation problem, and we must instead use numerical methods.

Update: Thanks to the comment of Nick, I will be more clear: The partial derivatives of the log-likelihood function for logistic regression are: $$ \frac{\partial}{\partial\beta_j}=\sum_{i=1}^n\left(y_i - \frac{1}{1+e^{-x_i^T\beta}}\right)x_{ij}, $$ where thus each partial derivative depends on $\beta$ in a highly non-linear way (as a sum of non-linear functions). The first-order condition of these set of partial derivatives, i.e. that $\frac{\partial}{\partial\beta_i}=0$ for all $i$, does not have an analytical solution. I do not have a reference to a proof, perhaps someone can add this is the comments. But it should be clear that this is a difficult problem.

This is the reason logistic regression does not have a closed form solution. The reason OLS has a closed form solution is because of the derivation I presented above. The OLS problem is in many ways more simple than that of logistic regression, and it is only because of the affinity of the gradient that it has a closed form solution. E.g., if you add a penalty to OLS, other than the ridge penalty, it will not lend itself to a closed form solution anymore.

$\endgroup$
  • $\begingroup$ The first paragraph is all good stuff but your answer to the question boils down there being no analytic solution for logit because there is no analytic solution. Fortunately, the issue has been discussed elsewhere. $\endgroup$ – Nick Cox Mar 5 '15 at 9:36
  • $\begingroup$ @NickCox, thank you very much for the great answer! $\endgroup$ – com Mar 5 '15 at 10:20
  • $\begingroup$ if you find the proof that first-order condition can not be solved analytically, please share with us. $\endgroup$ – com Mar 5 '15 at 10:21
  • $\begingroup$ I think it's @Tommy L who deserves the credit here. $\endgroup$ – Nick Cox Mar 5 '15 at 10:27

Not the answer you're looking for? Browse other questions tagged or ask your own question.