I am reading the following from the book Deep Learning, and I have the following questions.

enter image description here

  1. I don't quite understand second directional derivatives. The first directional derivative of a function $f:\mathbb{R}^m\to\mathbb{R}$ in the direction $u$ represents the slope of $f$ in the direction $u$. So what does the second directional derivative along the direction $u$ represent?
  2. In the above paragraph, I understood that $d^THd$, the second directional derivative of $f$ in the direction $d$ ($||d||_2=1$), is given by the corresponding eigenvalue when $d$ is an eigenvector of $H$, because if $d$ is an eigenvector of $H$ then $d^THd=d^T\lambda_d d=\lambda_d d^Td=\lambda_d$. However, I don't understand the statement "For other directions of $d$, the directional second derivative is a weighted average of all the eigenvalues, with weights between $0$ and $1$"::--Since $H$ is real symmetric, $H$ has $m$ independent orthogonal eigenvectors, which form a basis for $\mathbb{R}^m$. Thus, if $d$ is not an eigenvector, then $d=c_1x_1+\cdots +c_mx_m$ for some scalars $c_i$s and eigenvectors $x_i$s. Thus, $$d^THd=d^TH(c_1x_1+\cdots +c_mx_m)\\=d^T(c_1\lambda_1x_1+\cdots +c_m\lambda_mx_m)\\=c_1^2||x_1||^2\lambda_1 +\cdots +c_m^2||x_m||^2\lambda_m$$, which is ofcourse the weighted average of all the eigenvalues of $H$. But I don't understand why the weights lie between $0$ and $1$ as given. In fact, there is no reason to believe that the weights $c_i^2||x_i||^2$ to be in the range $(0,1)$.
  3. Also, I don't understand the statement "The maximum eigenvalue determines the maximum second derivative, and the minimum eigenvalue determines the minimum second derivative". Can you explain this?

1 Answer 1

  1. Think carefully about what you mean when you describe the directional derivative as the "slope" of a function $f$ in a certain direction. The concept of a "slope" only really makes sense in the context of a function whose domain is one-dimensional.

    It helps to think of it this way. Let $f({\bf x})$ be a scalar valued function whose domain is $\mathbb{R}^D$, and let ${\bf X}(t)$ be a continuous vector valued function parameterised in $\mathbb{R}^1.$ ${\bf X}(t)$ can be thought of as a path taken by a point particle in $D$-dimensional space. Then we can define $g(t) = f({\bf X}(t))$ as the value of the field $f$ experienced by the particle as it moves along the path. $g(t)$ is $\mathbb{R} \rightarrow \mathbb{R}.$ It's with this function that we can talk about ordinary one-dimensional concepts like slopes, second derivatives, etc. For example, if ${\bf X}(t_0) = {\bf u}$ is some direction, the way to think of the directional derivative of $f$ in direction $\bf u$ is $g'(t_0)$, i.e. the slope of $g$ at $t_0.$ The second directional derivative of $f$ in direction $\bf u$ is $g''(t_0).$

    Any time we talk about an $n$'th "directional derivative" of some function $f$ in $\mathbb{R}^D$ at point ${\bf x}_0$ in the direction $\bf u$, we're implicitly talking about the $n$'th derivative of a function in $\mathbb{R}^1$ described by the value of $f$ parameterized by a one-dimensional path in $\mathbb{R}^D$ whose direction at ${\bf x}_0$ is $\bf u$.

    Another (perhaps simpler) way of thinking about it is as follows. If $f({\bf x})$ is a scalar valued function whose domain is $\mathbb{R}^D$ then the directional derivative $D_{\bf u} f ({\bf x})$ is also a scalar valued function whose domain is $\mathbb{R}^D.$ The second directional derivative of $f$ in $\bf u$ is simply the directional derivative of $D_{\bf u} f ({\bf x}),$ i.e. it is $D_{\bf u} D_{\bf u} f ({\bf x}).$

  2. Remember, $d$ is assumed in the text to be a unit vector. If $d$ is a linear combination of unit vectors in an orthonormal basis given by $d = \sum_i c_i x_i$ then you can easily show that $d^T d = \sum_i c_i^2.$ Because $d$ is assumed to be a unit vector it follows that $\sum_i c_i^2 = 1.$ If $d$ is not specifically an eigenvector then none of the $c_i$'s are $1,$ which means that $c_i \in [0,1)$ $\forall i.$

    Here I'm assuming that the basis is normalized (i.e. $||x_i||^2 = 1$). However, you don't need to assume this. In general, the weight of each eigenvalue is simply $(c_i ||x_i||)^2$. If $d$ is a unit vector, given the orthogonality of the basis you can show that $\sum_i (c_i ||x_i||)^2 = 1.$

  3. This is a claim that the direction for which a function $f$ has the highest second derivative is the direction of the eigenvector of the Hessian of $f$ that has the highest eigenvalue. This claim follows from what was claimed above. If the directional second derivative of $f,$ for any directional vector $d,$ is the weighted average of all eigenvalues of the Hessian with weights given by $d$'s projection in each eigenvector, then obviously the direction that maximizes the directional derivative is the one for which the weight of the highest eigenvalue is $1,$ and all other weights is $0.$ (Recall the constraint that $\sum_i c_i^2 = 1.$) This direction is just the eigenvector that corresponds to the highest eigenvalue.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.