0
$\begingroup$

For example, an analytical solution is possible for linear regression (i.e. the normal equations) but it is not possible for logistic regression but logistic regression can be optimized with gradient descent. However, both solutions involve taking the gradient wrt to the weights. With linear regression the gradient is set to zero and we solve for w. But with logistic regression the gradient is used for updating the weights in gradient descent. So is the real reason that logistic regression cannot be solved analytically because it is impossible to solve for w if we set the gradient equal to zero?

$\endgroup$
4
  • 3
    $\begingroup$ correct: the logistic regression would result in a system of equations that is not linear, so you can't obtain the weights nicely like you can in a linear regression setting. So, instead, you maximize a likelihood using a numerical optimization procedure, one of which is gradient descent but there are other approaches also. It's been a long time (so memory is fuzzy) but John Fox's blue companion book (companion to his CAR text which is also good) has some nice material ( code and verbal ) on estimating the logistic numerically. $\endgroup$
    – mlofton
    May 9, 2019 at 0:01
  • 2
    $\begingroup$ @mlofton, post this as an answer? FWIW, iteratively reweighted least squares (IRLS) is probably the most common algorithm used for fitting logistic regression ... $\endgroup$
    – Ben Bolker
    May 9, 2019 at 0:19
  • 1
    $\begingroup$ @Ben Bolker: Yes, I think that's one of the methods that John uses in his book. Thanks for the name. If you want to make it an answer, that's fine. It doesn't matter to me one way or the other. or, if it's not allowed and you want me to, I can copy it to an answer ? I'm not familar with the rules so I don't know if I'm allowed to do this ? If I am, I'll do it. $\endgroup$
    – mlofton
    May 10, 2019 at 2:07
  • 1
    $\begingroup$ You're encouraged to answer your own question, if you can. $\endgroup$
    – Ben Bolker
    May 10, 2019 at 2:58

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.