Variance of evidence lower bound(ELBO) loss function

Question

When using Bayesian optimisation in a neural network our loss function is equal to:

Here the first term is the KL divergence between the approximate and true posteriors. The second term is the loglikelihood function.

1. I want to know how we can approximate the variance of the estimated parameters.
2. If the KL divergence is zero, does that mean we just go back to maximum likelihood optimisation, which means that the covariance matrix is equal to the negative inverse Hessian?
3. Can we also calculate it when the KL divergence is not zero?
4. Can we, in that case, add the hessians of both terms?

EDIT:

First of all thanks for taking the time to answer my question. You already cleared a lot of things up. However, is still don't completely get it so let me rephrase my question:

Let's say we simulate some data and fit a logistic regression model. The central limit theorem tells us that when we have enough data, the coefficient estimate is normally distributed with as mean the true value of the coefficient, and as variance the negative inverse of the hessian. Lets now fit a Bayesian neural network with one node and a sigmoid layer to this data. If we maximise the ELBO, will our neural network estimate have the same variance as the logistic regression model? Is there a way we can proof what the asymptotic properties of the neural network estimations will be?

Answer 1

I think there's a bit of a misunderstanding as to what we're doing in Bayesian neural networks.

We know there's a real posterior distribution over the weights given the data but that the distribution is intractable or too computationally expensive to compute. To solve this we're approximating this posterior distribution with some variational distribution with parameters we can control, for example a multivariate-normal with mean $\mu$ and covariance matrix $\Sigma$. So we choose $\mu$ and $\Sigma$ such that the KL divergence between the variational and posterior is minimised (Which is the equivalent of maximising the ELBO). It should then be clear that maximising the ELBO intrinsically gives you the variance of the parameters since it is approximating the whole distribution over the parameters, not just giving you a point estimate.

To answer your specific questions though:

1. Optimising the ELBO intrinsically approximates the variance of the parameters.

2. No, if the KL divergence is 0 then your approximate posterior is actually the exact posterior.

3. Yes, this is the case pretty much all of the time.

4. Not quite sure what you mean by this but a common technique is the Laplace approximation that makes use of the Hessian.

EDIT in response to question edit:

I suppose this is just a classic question on how different Bayesian and frequentist results are. The two approaches understand the parameters differently so it doesn't make much sense to consider the variance of the parameters because that variance represents completely different things under the different paradigms. In short though no, the variance will not be the same and will be different based both on your prior distribution and the variational distribution that you choose.

In terms of the asymptotics, that's not really my area, but I imagine there are results. Bayesian MAP estimators usually converge on the MLE for example.