is there a "generic" gradient descent

Question

on this week we learned that the general form of the update step for gradient descent is:

$x := x - \alpha \frac{\partial f}{\partial x}$

So, in order to find x where f is minimum, we have to calculate the derivative df/dx. Along this course, we found the analytical solution for dC/dw and substituted in the gradient descent formula to calculate the steps. This means that the update formula depends on the function we are trying to minimize. So this is my question: is there a generic version of gradient descent, so we don´t need to calculate the derivative analytically?

Answer 1

No, and it would not be logically coherent if there was one. The function for which you're ultimately calculating the gradient defines your optimization objective. In other words, it puts a number on how well you've answered the question.

Since there is no such thing as a generic problem, there is no such thing as a generic solution to it. The derivative form you've provided is the most general way to express your problem statement (or at least, it would be if you replaced it with the symbols for multidimensional gradients, but it's a nitpick).

Answer 2

You could use the numerical differentiation $$ \frac{df}{dx_i} = \lim_{h\rightarrow0}\frac{f(x_1,..., x_i+h,..., x_n) - f(x_1,..., x_i,..., x_n)}{h} $$ where you keep the original function as $$f(x_1, x_2, ... x_n).$$

This could of course be done in multivariate gradient. Just make sure you append the right variable in the right row.

Edit:

For a small step size h (in reality this is governed by the machine epsilon)

As the equation becomes more "difficult" this numerical solution for the derivative may become too expensive and the analytical solution would be more suitable.

Put the +h wrong