4
$\begingroup$

This question already has an answer here:

I'm trying to understand why, when trying to minimise an objective function, gradient descent is often used, rather than setting the gradient of the error to zero, and solving it analytically.

In school, I was taught that in order to minimise a function, you take it's first derivative with respect to the parameters, and then set this to zero. Solving this equation then yields one or more solutions (local minima), and you can try each one out to find which is the global minimum.

However, when learning about neural networks, I have seen that training is carried out by using gradient descent (or back-propagation in the case of multi-layer networks). Here, the gradient of the error function is taken, but instead of setting this to zero, you just move the parameters a small amount in the direction of this gradient.

I don't understand why the analytical approach isn't used, given that it has the obvious strength that the solution is perfect, and guarantees returning the global minimum, rather than potentially a local minimum as with gradient descent. My intuitive reason why the analytical approach isn't used is that it becomes very hard to solve an equation with all these parameters by hand, but, perhaps naively, I would have thought that there were automated ways to solve these kind of complex equations these days? Wouldn't the effort in enabling this be worth it, considering the better results, compared to the effort in getting gradient descent to work only to get inferior results?

Thank you :)

$\endgroup$

marked as duplicate by Sycorax, gung, Greenparker, Christoph Hanck, John May 18 '16 at 22:18

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

  • 1
    $\begingroup$ The same question is answered in the context of linear models here: stats.stackexchange.com/questions/23128/… $\endgroup$ – Sycorax May 18 '16 at 14:47
  • 1
    $\begingroup$ Do you have a good method to find the zeros of the gradient? $\endgroup$ – whuber May 18 '16 at 15:23
1
$\begingroup$

The system of equations you get setting the derivatives equal to zero cannot generally be solved analytically. For instance suppose I want choose $10>x>0$ to minimize $xln(x)-\sqrt{x}$ (which does attain a minimum in $10>x>0$) then our first order condition is $ln(x)+1-\frac{1}{2\sqrt{x}}=0$ which does not admit a closed form solution. Of course, we can try and find a solution using numerical methods, but that is precisely what gradient descent does!

$\endgroup$
  • 1
    $\begingroup$ +1 Good point. But please double-check your "first order condition": it does not equal the derivative of your objective function. Although it does not change the validity of your answer, note that this particular example does have an "analytical solution" in terms of the Lambert W function. $\endgroup$ – whuber May 18 '16 at 15:53
  • $\begingroup$ @whuber Whoops, thanks for pointing this out. I have fixed the FOC. Was not aware of the Lambert W function, will leave the comment as it is though to avoid confusion (as far as I understand what constitutes 'closed form' is somewhat context dependent anyway). $\endgroup$ – SecretlyAnEconomist May 18 '16 at 20:31
  • $\begingroup$ Indeed, MAPLE finds an analytical solution in terms of LambertW for the argmin and min objective function value using minimize(x*log(x)-sqrt(x),x=0..10,location); Or with solving for derivative = 0. $\endgroup$ – Mark L. Stone May 18 '16 at 20:57
  • 1
    $\begingroup$ @whuber The Lambert W function is so popular, that there's an upcoming 4 day conference celebrating 20 years since the first comprehensive survey of the Lambert W function was published www.apmaths.uwo.ca/~djeffrey/LambertW/LambertW.html . $\endgroup$ – Mark L. Stone Jun 1 '16 at 12:15

Not the answer you're looking for? Browse other questions tagged or ask your own question.