1
$\begingroup$

It seems to me that natural gradient is simply derived from directional derivative. For example, for a vector $v$,

$\tilde{\nabla} f \cdot v = G^{-1} \nabla f \cdot v = \lim_{h\to0} \frac{f(x+hv)-f(x)}{h}$

If they are equivalent, why do we need two terms for the same thing?

References:

Directional derivative (https://en.wikipedia.org/wiki/Directional_derivative)

Why Natural Gradient? (http://www.yaroslavvb.com/papers/amari-why.pdf)

Natural Gradient Works Efficiently in Learning (http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.452.7280&rep=rep1&type=pdf)

$\endgroup$
2
$\begingroup$

From the definition of the directional derivative

$$ Df(x, v) = \lim_{h\to0} \frac{f(x+hv)-f(x)}{h} $$

it is not obvious that $Df$ should have any special structure. On the other hand, the mapping

$$ v \mapsto \nabla f(x) \dot v $$

has a lot of obvious structure - it is a linear map.

The point of these two definitions is that the gradient exists. I.e., we can only have a thing called a gradient vector if the directional derivative is a linear map. It didn't have to be, we could have found ourselves in a universe where directional derivative have much worse behaviour, and mathematics would be a lot more difficult.

$\endgroup$
  • $\begingroup$ Sorry if I misunderstand. Your answer is explaining "gradient" and "directional derivative". I agree with what you said. But my original question is more about the "natural gradient" (the $\tilde \nabla f$) For me, there is only one gradient and we can derive it from the directional derivative. I don't know why we need to have gradient and natural gradient. $\endgroup$ – LKS Feb 22 '18 at 4:12
  • $\begingroup$ I have never heard of this distinction, even though I studied differential geometry in graduate school. What's the source for this terminology? $\endgroup$ – Matthew Drury Feb 22 '18 at 6:09
  • 1
    $\begingroup$ @MatthewDrury It's from information geometry and it's application to the natural gradient descent algorithm described by Amari in the paper "Natural Gradient Works Efficiently in Learning " that LKS cites above. Section 2 of this paper arxiv.org/pdf/1301.3584.pdf gives you a brief overview or if you can get a copy there's a longer summary of the area in Chapter 12 of Amari's 2016 book. $\endgroup$ – MachineEpsilon Feb 22 '18 at 7:21
  • 1
    $\begingroup$ @MatthewDrury "never heard of this distinction" that's my point. It seems to me that there is only one kind of "gradient" which is from the definition of directional derivative. I don't understand why the "natural gradient" paper needs to emphasize the distinction. To be clear, the notation in my equation above is $\tilde \nabla f$ is the "natural gradient" and $\nabla f$ is the "gradient" with Euclidean norm. $\endgroup$ – LKS Feb 22 '18 at 15:09
  • $\begingroup$ Thanks for the clarification LKS, I apologize for not understanding your question. I think a differential geometer would agree with you on that one. $\endgroup$ – Matthew Drury Feb 22 '18 at 15:18
2
$\begingroup$

I think the difference is a bit subtle, but natural gradients are suppose to be an intrinsic properties of the function $f$ and do not depend on the parameterisation of $f$. In particular, the intuition is we want to know the curvature of $f$ independently of a choice of coordinate system. Whereas directional derivatives as defined by your definition require specification of a vector $v$ and so also are explicitly calculated with respect to vector space and a coordinate system centered around $x$.

Natural gradients don't need a function $f$ in general to be specified, only a geometric structure (usually in this context a Riemannian manifold). To actually do any calculations we still need to choose a coordinate system. In that example of Amari's of calculating the Riemannian metric tensor $G$ when the geometric structure being studied is $\mathbb{R}^2$, to obtain an explicit form for $G$ we need to choose coordinates. Since we are studying $\mathbb{R}^2$ with respect to polar coordinates, the Riemannian metric tensor is not constant. To put it another way, polar coordinates introduce an artificial curvature on $\mathbb{R}^2$ even though the underlying space is intrinsically flat.

People with a background in differential geometry (like Amari) seem to really care about these coordinate-free, intrinsic definitions. Sometimes it is not possible to have global coordinate system for geometric objects that they study. My impression is that the geometric structures of distributions which arise in statistics there is often a natural and important global coordinate system.

$\endgroup$
  • $\begingroup$ That's also my intuition in the beginning, especially for the second part (directional derivative): when I work out the gradient in polar coordinate, I always have the mapping from euclidean coordinate in mind. $\endgroup$ – LKS Feb 22 '18 at 4:52
  • $\begingroup$ For the natural gradient, does the calculation of $G$ require knowing the function $f$? In "Why Natural Gradient?", the calculation of $G$ is done in Equation 15 and the function $f$ is not even defined yet. This really confuses me. $\endgroup$ – LKS Feb 22 '18 at 4:54
  • $\begingroup$ I have tried to elaborate on this in the answer but the structure being studied is not a function $f$ but rather $\mathbb{R}^2$. $\endgroup$ – MachineEpsilon Feb 22 '18 at 5:21
  • $\begingroup$ I really like "To actually do any calculations we still need to choose a coordinate system", "polar coordinates introduce an artificial curvature". Appreciate that @MachineEpsilon. For "to obtain an explicit form for $G$", it seems that we need the mapping between polar coord and euclidean coord. Why does the euclidean coord (always) appear and seem necessary? $\endgroup$ – LKS Feb 22 '18 at 15:27
  • $\begingroup$ "the geometric structures of distributions which arise in statistics there is often a natural and important global coordinate system." This sounds crucial. I am thinking about it. $\endgroup$ – LKS Feb 22 '18 at 15:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.