I was reading the deep learning book by Begnio, Goodfellow and Courville and there was one section where they explain the second derivative that I don't understand (section 4.31):
The second derivative tells us how the first derivative will change as we vary the input. This is important because it tells us whether a gradient step will cause as much of an improvement as we would expect based on the gradient alone.
the part in bold is what does not make sense to me.
I think I do understand what the second derivative means. The second derivative simply measures how much the gradient/tangent slope $f'(x)$ changes as we make small changes in $x$. i.e. how small changes in $x$ changes the gradient $f'(x)$. So for example, if we had a large second derivative a we made a tiny move, then the tangent line should change a lot. Thus it makes sense it measures the speed at which a tangent line becomes steep and thus why its sometimes referred as a measure of the curvature. That makes sense.
What does NOT make sense to me is that part in bold. How does it inform us at all about if a gradient step would make as much of an improvement as we would expect based on the gradient alone? I don't think I even understand what that means in english as a sentence. The main reason that it seems confusing to me is that a gradient step is independent of the second derivative, so a gradient step will change the target function whatever amount it has to. In fact the second derivative is the rate of change of the derivative and doesn't seem to hold direct info on the target function we are trying to optimize so I don't know what it means with:
gradient step will cause as much as an improvement as we would expect based on the gradient alone.
can someone explain me what this means?