I'm a math Ph.D. without formal training in statistics. Quite a few papers on normalization methods in deep learning mention the Fisher information matrix and how it's related to the Riemannian metric of the Riemannian manifold defined by the parameters of a neural network. It looks like this is closely related to natural gradients, which is another topic, I've never looked at.
Does anyone know of any good sources where I could actually read about this connection and which cover it in detail? I've seen quite a few handwavy accounts of this over the web, but I'm looking for something to help actually properly understand it. It wouldn't hurt to also get pointers to some code that uses this on a toy problem, so I could play around with it.