# If I set a Gaussian prior in the parameter space, can I use the variance as diagonal of the Hessian?

Say I have a neural network, where I put a Gaussian prior over the parameters $$\theta_i \sim N(\mu_i, \sigma_i^2)$$ and that I learn both $$\mu$$ and $$\sigma$$ via the reparameterization trick $$f = \mu + \epsilon \cdot \sigma$$

After training, I'll have for each weight its corresponding $$\mu, \sigma$$

My question is, can I consider $$\sigma$$ as a "proxy" for the diagonal of the Hessian?
In my mind it works because $$\sigma >>0$$ means that that parameter is pretty invariant to big changes, thus should have a second derivative pretty large, right?

• seems something related to variational approximation of the posterior, but not sure though Commented Feb 25 at 15:56
• Sounds like an interesting question; maybe you can try it out and let us know? (get the diagonal of the hessian efficiently in an autodiff framework by taking the derivative of the squared magnitude of the gradient). Commented Feb 25 at 16:15
• @JohnMadden you know what? i like your idea, i'll let you know asap Commented Feb 25 at 18:25
• @JohnMadden done, answered (though there is for sure a connection between variance and hessian, related by the Fisher Information, but it doesn't play a role in this case) Commented Feb 25 at 21:48

TLDR: no, the $$\sigma$$ converge to $$0$$

Well, seems like that an empirical test was more than enough to answer to this question

I trained a linear model on a test dataset with the following objective: $$L(\mu, \log\sigma) = N^{-1}\sum_{i=0}^N (\beta x - y ), \\ \beta = \mu + \epsilon \cdot \exp(\log\sigma)\\ \epsilon \sim N(0,1)$$

And the result is the following:

Clearly, the loss converges, but the $$\log\sigma\rightarrow 0$$

A colleague of mine correctly pointed out that the "variance" term will just be white noise, so half a time will make the prediction better, and half not.

Indeed, this trick is used to train VAEs, where in addition to it, there is also an anchor to a prior (the loss is $$L = L' + D_{KL}(p_\theta(z)) || q(z)$$) thus forcing $$\sigma$$ to be similar to something specific

• +1 awesome work, and those plots beautifully illustrate your point. Commented Feb 25 at 21:58