# How does the second derivative inform an update step in Gradient Descent?

I was reading the deep learning book by Begnio, Goodfellow and Courville and there was one section where they explain the second derivative that I don't understand (section 4.31):

The second derivative tells us how the first derivative will change as we vary the input. This is important because it tells us whether a gradient step will cause as much of an improvement as we would expect based on the gradient alone.

the part in bold is what does not make sense to me.

I think I do understand what the second derivative means. The second derivative simply measures how much the gradient/tangent slope $f'(x)$ changes as we make small changes in $x$. i.e. how small changes in $x$ changes the gradient $f'(x)$. So for example, if we had a large second derivative a we made a tiny move, then the tangent line should change a lot. Thus it makes sense it measures the speed at which a tangent line becomes steep and thus why its sometimes referred as a measure of the curvature. That makes sense.

What does NOT make sense to me is that part in bold. How does it inform us at all about if a gradient step would make as much of an improvement as we would expect based on the gradient alone? I don't think I even understand what that means in english as a sentence. The main reason that it seems confusing to me is that a gradient step is independent of the second derivative, so a gradient step will change the target function whatever amount it has to. In fact the second derivative is the rate of change of the derivative and doesn't seem to hold direct info on the target function we are trying to optimize so I don't know what it means with:

gradient step will cause as much as an improvement as we would expect based on the gradient alone.

can someone explain me what this means?

• i.stack.imgur.com/kGmcb.png – Glen_b Nov 23 '17 at 3:05
• @Glen_b sorry for being dense, but what is that suppose to mean? :) – Pinocchio Nov 23 '17 at 3:22
• For each step we're minimizing in the direction of the gradient, so in effect we have a univariate problem. So lets look at one! Imagine you're sitting at the far right, and trying to descend. If the the gradient is correctly telling how much we'll descend for each unit of movement to the left, we'll follow the function down the black line as we step along. But if the second derivative is positive we actually follow something like the blue curve and the gradient is overly optimistic about how much we gain unless we only took a tiny step (conversely red shows a locally negative 2nd derivative) – Glen_b Nov 23 '17 at 6:32

I agree with your distaste for the writing. It seems as though you have an understanding of what is going on, but I will attempt to clarify why the second derivative is important. Consider a two-dimensional orthogonal system. Since they are orthogonal we can look at them independently, and together. This need not be the case, but I use the orthogonal system to avoid the linear algebra which may muddy the intuition .

In the $x_1$ dimension, the objective, f, varies roughly as $f=x_1^2$. In the $x_2$ dimension, the objective varies as $f = .00001x_2^2$. The minima is f(0,0) = 0.

This is the gradient descent update in each dimension:

• $x_{1,k+1} = x_{1,k} - 2\alpha x_{1,k}$

• $x_{2,k+1} = x_{2,k} - .00002\alpha x_{2,k}$

Where $\alpha$ is the learning rate. That is, According to a gradient descent update, if you start at about (1,1) then after a few iterations you will be at $\approx (0,1)$ because the gradient in the $x_2$ direction is already very near zero. True, we may have predicted this based on the fact that the gradient at every point in the $x_2$ direction is near zero, but it is still undesirable -- I think that is the point they were trying to make in the bolded sentence.

Now we note that $\frac{\partial^2 f}{\partial x_2^2} = .00002$. $\frac{\partial^2 f}{\partial x_1^2} = 2$. Dividing by this amounts to accounting for the curvature (or lack thereof) in that dimension. Now let's solve these two 1-d functions using second order information. Recall that the form is $x_{k+1} = x_k - \alpha \frac{\partial^2 f}{\partial x_k^2}^{-1} \frac{\partial f}{\partial x_k}$:

• $x_{1,k+1} = x_{1,k} - \alpha x_{1,k}$

• $x_{2,k+1} = x_{2,k} - \alpha x_{2,k}$

That is, they are converging at the same rate, exactly as we would hope they would!

• beatiful! that is super intuitive. Essentially in this case its really clear how dividing by the second derivative makes the two updates for each dimension behave the same. What seems like magic is that the second derivative has such an effect...wonder why! – Pinocchio Nov 23 '17 at 0:32
• The difference between these two functions is solely in the curvature -- once you divide by the inverse of that curvature you have negated that difference. The effect is startlingly practical in this case. In cases which the dimensions are not orthogonal (all real cases) we would have to invert a Hessian which comes with computational difficulties. Furthermore, if it is not a linear system then we would have to calculate a Hessian inverse at each step! Advances in this direction are cutting edge in the optimization literature – David Kozak Nov 23 '17 at 0:37
• I guess what seems like magic to me is that division worked to "cancel out the effect of the bad curvature" and not something else like subtraction. Nearly looks like a coincidence! – Pinocchio Nov 23 '17 at 0:55
• Agreed :) I was astounded when the intuition of it clicked for me as well. – David Kozak Nov 23 '17 at 0:56

You expectation might be that for a small change of the coordinates from $$s= x_{n+1} - x_n$$ the change of the function is approximately linear $$f(x_{n+1}) \approx f(x_n) + \nabla f(x) (x_{n+1}-x_{n})$$

Whether the improvement will be as much of an improvement as the one based on such linear approximation depends on the second derivative.

If the second directional derivative (along the direction of change $$s$$) is negative then you will get effectively a smaller change in the function.

E.g. in one dimensional quadratic function $$f(x)=ax^2 + bx +c$$ you have

$$f(x_{n+1}) = f(x_{n}) + f^\prime(x_n) (x_{n+1}-x_{n}) + \frac{f^{\prime\prime}(x_n)}{2}(x_{n+1}-x_{n})^2$$

More generally, instead of just the second order term you have a Taylor expansion

$$f(x_{n+1}) = f(x_{n}) + \sum_{k=1}^\infty \frac{f^{(k)}(x_n)}{k!} (x_{n+1}-x_{n})^k$$

although in many practical cases for small changes $$(x_{n+1}-x_{n}) \ll 1$$ the higher order terms are sufficiently small or even vanish (e.g. polynomial functions).

Alternatively you could also express:

$$f(x_{n+1}) = f(x_{n}) + \int_{x_n}^{x_{n+1}} f^\prime(s) ds$$

and

$$\min \left[ f^\prime(s) \right] (x_{n+1}-x_{n}) \leq \int_{x_n}^{x_{n+1}} f^\prime(s) ds \leq \max \left[ f^\prime(s) \right] (x_{n+1}-x_{n})$$

When $$\vert f^{\prime \prime} \vert$$ is small between $$x_{n}$$ and $$x_{n+1}$$, then the minimum $$f^\prime$$ and maximum $$f^\prime$$ values won't differ much.

Second derivatives are used to understand the rate of change of derivatives. Considering the huge number of hyper-parameters involved in building the trained model, it is always necessary to detect early the accuracy rate of the model being trained. Many of us, spend a considerable time in training the models with different set of hyper-parameters to reach an optimal solution. Thus, early detection of the learning trend helps in saving resources. Second derivatives are one of the easiest ways to help us in early detection of the learning trends and provides the model designer a feedback to intervene and refine the hyper-parameters at a much early stage, thus saving precious time and other resources.