The method of natural gradient adaption has been proposed as an improvement on gradient descent (e.g. Amari, "Natural Gradient Works Efficiently in Learning", 1998.) In gradient descent the usual update step is $$ w_{k+1} = w_k - \mu_k \frac{\partial J(w_k)}{\partial w_k} $$ where $w_k$ is the weights parameter at iteration $k$, $\mu_k$ is the step size, and $J$ is the cost function. The method of natural gradients replacement proposes to replace it with the following: $$ w_{k+1} = w_k - \mu_k G^{-1}w_k \frac{\partial J(w_k)}{\partial w_k} $$ where $ G^{-1}w_k$ is a matrix that contains information on the direction of steepest ascent (the Riemannian metric tensor for the manifold of parameters).

This seems very nice in theory, but outside of simple cases it's hard to see how you could calculate $G^{-1}w_k$. In practice, it seems like the cases where $G$ is effective calculable would only be when you had a nice distribution which was a member of the exponential family. But if you're fitting a model in such a family, then my understanding is that you are better off using Newton's method or a quasi-Newton method. Are there situations where natural gradients are actually computed in practice?

  • 2
    $\begingroup$ They're used in stochastic variational inference (columbia.edu/~jwp2128/Papers/HoffmanBleiWangPaisley2013.pdf), but it turns out that you don't actually have to compute them in practice (from what I remember), see section 2.3. $\endgroup$ – aleshing Feb 22 '18 at 4:10
  • $\begingroup$ Thank you, that's interesting. I'll have a look at the reference. $\endgroup$ – MachineEpsilon Feb 26 '18 at 5:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.