Fisher information vs Posterior Covariance I have a parameter $\theta$ and data $y = f(\theta) + \mathrm{noise}$. My goal is finding the best fit for $\theta$ and assess the uncertainty I have on this best fit.
I see two competing approaches for doing this:

*

*I can compute the MLE $\widehat \theta =\arg \max_\theta p(y \mid \theta)$ with some optimization algorithm and assess the reliability of the estimator, using the Fisher information $I(\theta) =- \nabla^2_\theta p(y \mid \theta)$, and evaluating its inverse $I^{-1}(\widehat \theta)$ at the MLE. This would be a good option, since I can evaluate the Hessian analytically in my case, and finding the MLE is easy with some Newton algorithm.

*I can generate a sample $\{\theta^{(i)}\}_{i=1,\ldots, N}$ from $p(y \mid \theta)$, e.g., with MCMC sampling. I can then study the empirical covariance matrix of the sample to determine correlations, uncertainties, etc.

For a specific dataset and a eight-dimensional parameter, I get the following results:


There seems to be a pattern here: the two matrices seem to be a multiple of each other. I struggle to understand to which extent this is true, and how the two approaches above are connected / distinct.
 A: MCMC is a computational technique for implementing Bayesian inference.
Normally, the difference between MLE and Bayesian inference is that Bayesian inference incorporates prior information. In this case, however, you appear to have omitted steps from MLE and Bayes so that both they collapse down to the same thing.
First, you have not actually computed Fisher information but rather observed information for the MLE estimate.
Computing the Hessian of the log-likelihood at the MLE gives observed information.
Fisher information requires the expected value of the Hessian over the distribution of $y$.
Secondly, and more importantly, you don't mention any prior information for the Bayes MCMC approach. By ignoring the prior, you are effectively assuming a uniform improper prior for $\theta$, in which case Bayes and MLE become virtually the same thing.
The agreement between the two approaches becomes more complete because you are using observed rather than Fisher information for the MLE, and observed information is what is used in Bayesian inference.
So the two covariance matrices you estimate at the end are presumably not just multiples of one another but exactly the same, modulo MCMC convergence.
A: Both the emperical Fisher information matrix and simulation are approximations. Because the methods approximate the same thing the results will look similar, but there are differences.
Emperical Fisher matrix

*

*the estimation uses the observed maximum likelihood estimate instead of the true value of the parameters

*the Fisher information matrix gives a lower bound for the variance, and depending on the distribution, the actual variance is only asymptotically equal to this lower bound

Simulations

*

*the estimation uses the observed maximum likelihood estimate instead of the true value of the parameters

*the sampling has a statistical variation and is not like an exact computation

