Are natural gradients ever computed in practice? - Cross Validated most recent 30 from stats.stackexchange.com 2019-08-23T05:35:45Z https://stats.stackexchange.com/feeds/question/329934 http://www.creativecommons.org/licenses/by-sa/3.0/rdf https://stats.stackexchange.com/q/329934 2 Are natural gradients ever computed in practice? MachineEpsilon https://stats.stackexchange.com/users/17760 2018-02-22T02:28:20Z 2018-02-26T04:57:43Z <p>The method of natural gradient adaption has been proposed as an improvement on gradient descent (e.g. <a href="http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.452.7280&amp;rep=rep1&amp;type=pdf" rel="nofollow noreferrer">Amari, "Natural Gradient Works Efficiently in Learning", 1998.</a>) In gradient descent the usual update step is $$w_{k+1} = w_k - \mu_k \frac{\partial J(w_k)}{\partial w_k}$$ where $w_k$ is the weights parameter at iteration $k$, $\mu_k$ is the step size, and $J$ is the cost function. The method of natural gradients replacement proposes to replace it with the following: $$w_{k+1} = w_k - \mu_k G^{-1}w_k \frac{\partial J(w_k)}{\partial w_k}$$ where $G^{-1}w_k$ is a matrix that contains information on the direction of steepest ascent (the Riemannian metric tensor for the manifold of parameters). </p> <p>This seems very nice in theory, but outside of simple cases it's hard to see how you could calculate $G^{-1}w_k$. In practice, it seems like the cases where $G$ is effective calculable would only be when you had a nice distribution which was a member of the exponential family. But if you're fitting a model in such a family, then my understanding is that you are better off using Newton's method or a quasi-Newton method. Are there situations where natural gradients are actually computed in practice? </p>