Logistic regression iterative process I don't understand something very fundamental about Logistic Regression.
For Linear Regression and for Naive Bayes there is a closed form for calculating parameters.
For some reason, logit regression doesn't have such a form. Do you know why?
 A: In the case of OLS, $f(\beta)=\frac12\|y - X\beta\|_2^2$, the gradient is $\nabla f(\beta)=X^T(X\beta - y)$. The first order condition is $\nabla f(\beta)=0$. Thus, at the optimum, $X^T(X\beta - y)=X^TX\beta - X^Ty = 0$. Solve for $\beta$ as $\beta=(X^TX)^{-1}X^Ty$. Thus, we are able to analytically solve for the optimum.
This is not the case for logistic regression, i.e. there is no analytic solution for the optimisation problem, and we must instead use numerical methods.
Update:
Thanks to the comment of Nick, I will be more clear:
The partial derivatives of the log-likelihood function for logistic regression are:
$$
\frac{\partial}{\partial\beta_j}=\sum_{i=1}^n\left(y_i - \frac{1}{1+e^{-x_i^T\beta}}\right)x_{ij},
$$
where thus each partial derivative depends on $\beta$ in a highly non-linear way (as a sum of non-linear functions). The first-order condition of these set of partial derivatives, i.e. that $\frac{\partial}{\partial\beta_i}=0$ for all $i$, does not have an analytical solution. I do not have a reference to a proof, perhaps someone can add this is the comments. But it should be clear that this is a difficult problem.
This is the reason logistic regression does not have a closed form solution. The reason OLS has a closed form solution is because of the derivation I presented above. The OLS problem is in many ways more simple than that of logistic regression, and it is only because of the affinity of the gradient that it has a closed form solution. E.g., if you add a penalty to OLS, other than the ridge penalty, it will not lend itself to a closed form solution anymore.
