2
$\begingroup$

on this week we learned that the general form of the update step for gradient descent is:

$x := x - \alpha \frac{\partial f}{\partial x}$

So, in order to find x where f is minimum, we have to calculate the derivative df/dx. Along this course, we found the analytical solution for dC/dw and substituted in the gradient descent formula to calculate the steps. This means that the update formula depends on the function we are trying to minimize. So this is my question: is there a generic version of gradient descent, so we don´t need to calculate the derivative analytically?

$\endgroup$
2
  • 1
    $\begingroup$ Unsure I've got your question. The formula you've posted is the "generic" gradient descent. Then you apply this generic formula to a specific function, and compute df/fx. You always need to calculate the derivative analytically to apply gradient descent. $\endgroup$
    – Soltius
    Jan 7 '20 at 14:21
  • 1
    $\begingroup$ often the derivative is not analytically calculated. look up "automatic differentiation". $\endgroup$
    – shimao
    Jan 7 '20 at 17:40
0
$\begingroup$

No, and it would not be logically coherent if there was one. The function for which you're ultimately calculating the gradient defines your optimization objective. In other words, it puts a number on how well you've answered the question.

Since there is no such thing as a generic problem, there is no such thing as a generic solution to it. The derivative form you've provided is the most general way to express your problem statement (or at least, it would be if you replaced it with the symbols for multidimensional gradients, but it's a nitpick).

$\endgroup$
1
  • $\begingroup$ The original question is whether there is a method which doesn't rely on analytical derivatives, not which is independent of the function entirely. This happens all the time (see autodifferentiation, numerical methods, etc). $\endgroup$
    – combo
    Jan 21 '20 at 22:48
0
$\begingroup$

You could use the numerical differentiation $$ \frac{df}{dx_i} = \lim_{h\rightarrow0}\frac{f(x_1,..., x_i+h,..., x_n) - f(x_1,..., x_i,..., x_n)}{h} $$ where you keep the original function as $$f(x_1, x_2, ... x_n).$$

This could of course be done in multivariate gradient. Just make sure you append the right variable in the right row.

Edit:

For a small step size h (in reality this is governed by the machine epsilon)

As the equation becomes more "difficult" this numerical solution for the derivative may become too expensive and the analytical solution would be more suitable.

Put the +h wrong

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.