I'm just learning about optimization, and the difference between an analytical solution, and a numerical one. Suppose there is a cost function
f(x), and we want to find the value of
x which minimizes this. In an analytical solution, we would differentiate with respect to
x, i.e. find
df(x)/dx. Whereas in a numerical solution, we would try values of
x, and see in which direction we need to change
x in order to move towards the solution (gradient descent).
It seems to me that the analytical solution is always preferred because it gives you an exact answer. You do not get stuck in local minima and you can be sure that your solution is correct. So why do we not always use analytical solutions?
I understand that numerical solutions make things easier because you don't have to work out by hand the analytical solution, but it seems to me that solving something by hand to get a precise answer is highly preferred to doing it computationally for an approximate answer. Can somebody clear this up for me? Thanks!