10
$\begingroup$

I'm trying to understand why, when trying to minimise an objective function, gradient descent is often used, rather than setting the gradient of the error to zero, and solving it analytically.

In school, I was taught that in order to minimise a function, you take it's first derivative with respect to the parameters, and then set this to zero. Solving this equation then yields one or more solutions (local minima), and you can try each one out to find which is the global minimum.

However, when learning about neural networks, I have seen that training is carried out by using gradient descent (or back-propagation in the case of multi-layer networks). Here, the gradient of the error function is taken, but instead of setting this to zero, you just move the parameters a small amount in the direction of this gradient.

I don't understand why the analytical approach isn't used, given that it has the obvious strength that the solution is perfect, and guarantees returning the global minimum, rather than potentially a local minimum as with gradient descent. My intuitive reason why the analytical approach isn't used is that it becomes very hard to solve an equation with all these parameters by hand, but, perhaps naively, I would have thought that there were automated ways to solve these kind of complex equations these days? Wouldn't the effort in enabling this be worth it, considering the better results, compared to the effort in getting gradient descent to work only to get inferior results?

Thank you :)

Improve this question
$\endgroup$
2
  • 2
    $\begingroup$ The same question is answered in the context of linear models here: stats.stackexchange.com/questions/23128/… $\endgroup$
    – Sycorax ♦
    May 18, 2016 at 14:47
  • 2
    $\begingroup$ Do you have a good method to find the zeros of the gradient? $\endgroup$
    – whuber ♦
    May 18, 2016 at 15:23

1 Answer 1

Reset to default
5
$\begingroup$

The system of equations you get setting the derivatives equal to zero cannot generally be solved analytically. For instance suppose I want choose $10>x>0$ to minimize $xln(x)-\sqrt{x}$ (which does attain a minimum in $10>x>0$) then our first order condition is $ln(x)+1-\frac{1}{2\sqrt{x}}=0$ which does not admit a closed form solution. Of course, we can try and find a solution using numerical methods, but that is precisely what gradient descent does!

Improve this answer
$\endgroup$
6
  • 1
    $\begingroup$ +1 Good point. But please double-check your "first order condition": it does not equal the derivative of your objective function. Although it does not change the validity of your answer, note that this particular example does have an "analytical solution" in terms of the Lambert W function. $\endgroup$
    – whuber ♦
    May 18, 2016 at 15:53
  • $\begingroup$ @whuber Whoops, thanks for pointing this out. I have fixed the FOC. Was not aware of the Lambert W function, will leave the comment as it is though to avoid confusion (as far as I understand what constitutes 'closed form' is somewhat context dependent anyway). $\endgroup$
    – SecretlyAnEconomist
    May 18, 2016 at 20:31
  • $\begingroup$ Indeed, MAPLE finds an analytical solution in terms of LambertW for the argmin and min objective function value using minimize(x*log(x)-sqrt(x),x=0..10,location); Or with solving for derivative = 0. $\endgroup$
    – Mark L. Stone
    May 18, 2016 at 20:57
  • 1
    $\begingroup$ @whuber The Lambert W function is so popular, that there's an upcoming 4 day conference celebrating 20 years since the first comprehensive survey of the Lambert W function was published www.apmaths.uwo.ca/~djeffrey/LambertW/LambertW.html . $\endgroup$
    – Mark L. Stone
    Jun 1, 2016 at 12:15
  • $\begingroup$ How do you prove that something doesn’t have a closed form solution? $\endgroup$
    – Pro Q
    Oct 30, 2022 at 15:55

Not the answer you're looking for? Browse other questions tagged or ask your own question.