Second directional derivate and Hessian matrix

Question

I am reading the following from the book Deep Learning, and I have the following questions.

1. I don't quite understand second directional derivatives. The first directional derivative of a function $f:\mathbb{R}^m\to\mathbb{R}$ in the direction $u$ represents the slope of $f$ in the direction $u$. So what does the second directional derivative along the direction $u$ represent?
2. In the above paragraph, I understood that $d^THd$, the second directional derivative of $f$ in the direction $d$ ($||d||_2=1$), is given by the corresponding eigenvalue when $d$ is an eigenvector of $H$, because if $d$ is an eigenvector of $H$ then $d^THd=d^T\lambda_d d=\lambda_d d^Td=\lambda_d$. However, I don't understand the statement "For other directions of $d$, the directional second derivative is a weighted average of all the eigenvalues, with weights between $0$ and $1$"::--Since $H$ is real symmetric, $H$ has $m$ independent orthogonal eigenvectors, which form a basis for $\mathbb{R}^m$. Thus, if $d$ is not an eigenvector, then $d=c_1x_1+\cdots +c_mx_m$ for some scalars $c_i$s and eigenvectors $x_i$s. Thus, $$d^THd=d^TH(c_1x_1+\cdots +c_mx_m)\\=d^T(c_1\lambda_1x_1+\cdots +c_m\lambda_mx_m)\\=c_1^2||x_1||^2\lambda_1 +\cdots +c_m^2||x_m||^2\lambda_m$$, which is ofcourse the weighted average of all the eigenvalues of $H$. But I don't understand why the weights lie between $0$ and $1$ as given. In fact, there is no reason to believe that the weights $c_i^2||x_i||^2$ to be in the range $(0,1)$.
3. Also, I don't understand the statement "The maximum eigenvalue determines the maximum second derivative, and the minimum eigenvalue determines the minimum second derivative". Can you explain this?

Answer 1

1. Think carefully about what you mean when you describe the directional derivative as the "slope" of a function $f$ in a certain direction. The concept of a "slope" only really makes sense in the context of a function whose domain is one-dimensional.

   It helps to think of it this way. Let $f({\bf x})$ be a scalar valued function whose domain is $\mathbb{R}^D$, and let ${\bf X}(t)$ be a continuous vector valued function parameterised in $\mathbb{R}^1.$ ${\bf X}(t)$ can be thought of as a path taken by a point particle in $D$-dimensional space. Then we can define $g(t) = f({\bf X}(t))$ as the value of the field $f$ experienced by the particle as it moves along the path. $g(t)$ is $\mathbb{R} \rightarrow \mathbb{R}.$ It's with this function that we can talk about ordinary one-dimensional concepts like slopes, second derivatives, etc. For example, if ${\bf X}(t_0) = {\bf u}$ is some direction, the way to think of the directional derivative of $f$ in direction $\bf u$ is $g'(t_0)$, i.e. the slope of $g$ at $t_0.$ The second directional derivative of $f$ in direction $\bf u$ is $g''(t_0).$

   Any time we talk about an $n$'th "directional derivative" of some function $f$ in $\mathbb{R}^D$ at point ${\bf x}_0$ in the direction $\bf u$, we're implicitly talking about the $n$'th derivative of a function in $\mathbb{R}^1$ described by the value of $f$ parameterized by a one-dimensional path in $\mathbb{R}^D$ whose direction at ${\bf x}_0$ is $\bf u$.

   Another (perhaps simpler) way of thinking about it is as follows. If $f({\bf x})$ is a scalar valued function whose domain is $\mathbb{R}^D$ then the directional derivative $D_{\bf u} f ({\bf x})$ is also a scalar valued function whose domain is $\mathbb{R}^D.$ The second directional derivative of $f$ in $\bf u$ is simply the directional derivative of $D_{\bf u} f ({\bf x}),$ i.e. it is $D_{\bf u} D_{\bf u} f ({\bf x}).$

2. Remember, $d$ is assumed in the text to be a unit vector. If $d$ is a linear combination of unit vectors in an orthonormal basis given by $d = \sum_i c_i x_i$ then you can easily show that $d^T d = \sum_i c_i^2.$ Because $d$ is assumed to be a unit vector it follows that $\sum_i c_i^2 = 1.$ If $d$ is not specifically an eigenvector then none of the $c_i$'s are $1,$ which means that $c_i \in [0,1)$ $\forall i.$

   Here I'm assuming that the basis is normalized (i.e. $||x_i||^2 = 1$). However, you don't need to assume this. In general, the weight of each eigenvalue is simply $(c_i ||x_i||)^2$. If $d$ is a unit vector, given the orthogonality of the basis you can show that $\sum_i (c_i ||x_i||)^2 = 1.$

3. This is a claim that the direction for which a function $f$ has the highest second derivative is the direction of the eigenvector of the Hessian of $f$ that has the highest eigenvalue. This claim follows from what was claimed above. If the directional second derivative of $f,$ for any directional vector $d,$ is the weighted average of all eigenvalues of the Hessian with weights given by $d$'s projection in each eigenvector, then obviously the direction that maximizes the directional derivative is the one for which the weight of the highest eigenvalue is $1,$ and all other weights is $0.$ (Recall the constraint that $\sum_i c_i^2 = 1.$) This direction is just the eigenvector that corresponds to the highest eigenvalue.