The derivative of Log likelihood function of logistic regression with respect to theta is

enter image description here

  1. Why can't we equate it to zero and solve for theta so that we can obtain a 'closed form solution' for theta? Is it because of the non linearity of theta in the equation?

  2. Can OLS method in linear regression which is a closed form solution, be categorized as a 'linear optimization' technique? (unlike gradient descent which is a 'non-linear optimization' method?)

  1. Indeed, when using a binomial logistic model, the estimate is the solution of the equation : $$\sum_{i}[y_i - \sigma(\theta^T \boldsymbol{x}_i)] \boldsymbol{x}_i = 0$$ Unfortunately (you can try), this equation is not solvable, that's why it's said that there is no closed form solution for $\theta$. One must use optimisation technique to numerically approximate a solution (for logistic regression, Newton-Raphson algorithm works fine since likelihood is concave). Non linearity does complicate the equation, but there are some estimators which are closed form and solution of non linear equations, for example the median solves the non linear equation : $\sum_i [\mathbb{1}_{x_i > \theta} - \mathbb{1}_{x_i < \theta}] = 0$. Linear estimating equations are solvable but there are not the only ones.

  2. I am not sure about the question. Indeed the estimating equation of logistic regression is non linear in $\theta$ and the estimating equation of an OLS is linear (and thus easily solvable). But if you refer to linear programming (LP) by "linear optimization", then OLS is not a LP, since the solution of OLS equation is a minimizer of a quadratic equation. OLS is a quadratic programming (QP). Be carefull not to confuse "linear estimating equation" with linear programming.

  • $\begingroup$ I was going to ask a similar question and then saw this thread. So for logistic regression we got nothing analogous to using pseudo-inverse matrix for solving least-squares problem for finding the coefficients of a polynomial? Do we need to use numerical optimization tools? $\endgroup$ – SomethingSomething Dec 10 '20 at 12:24
  • $\begingroup$ I didn't understand what you mean by ".using pseudo-inverse .. finding the coefficients of a polynomial". But yes, we need to use numerical optimization tools: Newton-Raphson method (or Iterative reweighted least square), BFGS, Nelder-Mead,... $\endgroup$ – Pohoua Dec 14 '20 at 9:35
  • $\begingroup$ Probably my knowledge is not based so well, so I don't describe things accurately. I said it because AFAIK pseudo-inverse matrix gives you the least-squares solution for a linear equation, which can be used for finding the coefficients of a polynomial for curve fitting ("regression"). I hoped there would be a similar (non-iterative) solution also for logistic regression. Thank you for your answer. $\endgroup$ – SomethingSomething Dec 14 '20 at 12:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.