Even in the case of, say, linear models, where you have an analytical solution, it may still be best to use such an iterative solver.

As an example, if we consider linear regression, the explicit solution requires inverting a matrix which has complexity O(N3). This becomes prohibitive in the context of big data.

Also, a lot of problems in machine learning are convex, so using gradients ensure that we will get to the extrema.

As already pointed out, there are still relevant non-convex problems, like neural networks, where gradient methods (backpropagation) provide an efficient solver. Again this is specially relevant for the case of deep learning.

Even in the case of, say, linear models, where you have an analytical solution, it may still be best to use such an iterative solver.

As an example, if we consider linear regression, the explicit solution requires inverting a matrix which has complexity $O(N^3)$. This becomes prohibitive in the context of big data.

Also, a lot of problems in machine learning are convex, so using gradients ensure that we will get to the extrema.

As already pointed out, there are still relevant non-convex problems, like neural networks, where gradient methods (backpropagation) provide an efficient solver. Again this is specially relevant for the case of deep learning.