If I set a Gaussian prior in the parameter space, can I use the variance as diagonal of the Hessian?

Question

Say I have a neural network, where I put a Gaussian prior over the parameters $\theta_i \sim N(\mu_i, \sigma_i^2)$ and that I learn both $\mu$ and $\sigma$ via the reparameterization trick $f = \mu + \epsilon \cdot \sigma$

After training, I'll have for each weight its corresponding $\mu, \sigma$

My question is, can I consider $\sigma$ as a "proxy" for the diagonal of the Hessian?
In my mind it works because $\sigma >>0$ means that that parameter is pretty invariant to big changes, thus should have a second derivative pretty large, right?

Answer 1

TLDR: no, the $\sigma$ converge to $0$

---

Well, seems like that an empirical test was more than enough to answer to this question

I trained a linear model on a test dataset with the following objective: $$ L(\mu, \log\sigma) = N^{-1}\sum_{i=0}^N (\beta x - y ), \\ \beta = \mu + \epsilon \cdot \exp(\log\sigma)\\ \epsilon \sim N(0,1) $$

And the result is the following:

Clearly, the loss converges, but the $\log\sigma\rightarrow 0$

A colleague of mine correctly pointed out that the "variance" term will just be white noise, so half a time will make the prediction better, and half not.

Indeed, this trick is used to train VAEs, where in addition to it, there is also an anchor to a prior (the loss is $L = L' + D_{KL}(p_\theta(z)) || q(z)$) thus forcing $\sigma$ to be similar to something specific