I know that gradient descent takes steps towards a minimum, but I am having trouble coming up with intuitions about when it will converge.
For example, on any given convex function is gradient descent guaranteed to converge? I'm inclined to say no because the steps could be too big, but I'm not certain.
More specifically, with ordinary least squares is gradient descent guaranteed to converge? I'm inclined to say no for the same reason, but again I'm not sure.