When using Bayesian optimisation in a neural network our loss function is equal to:
Here the first term is the KL divergence between the approximate and true posteriors. The second term is the loglikelihood function.
- I want to know how we can approximate the variance of the estimated parameters.
- If the KL divergence is zero, does that mean we just go back to maximum likelihood optimisation, which means that the covariance matrix is equal to the negative inverse Hessian?
- Can we also calculate it when the KL divergence is not zero?
- Can we, in that case, add the hessians of both terms?
First of all thanks for taking the time to answer my question. You already cleared a lot of things up. However, is still don't completely get it so let me rephrase my question:
Let's say we simulate some data and fit a logistic regression model. The central limit theorem tells us that when we have enough data, the coefficient estimate is normally distributed with as mean the true value of the coefficient, and as variance the negative inverse of the hessian. Lets now fit a Bayesian neural network with one node and a sigmoid layer to this data. If we maximise the ELBO, will our neural network estimate have the same variance as the logistic regression model? Is there a way we can proof what the asymptotic properties of the neural network estimations will be?