Hi guys I am doing some model selection and need to estmiate both observed and expected Fisher Information Matrix to calculate this link. How do you estimate that expected one numerically?
I find it easiest to understand these things if I first consider a model with a single parameter, call it $\theta$. Typically we want to choose $\theta$ so as to maximize the likelihood (i.e. conditional probability) of the data we observe. Actually, it's more useful to maximize the log of the likelihood, so that rather than multiplying the likelihoods of all the observed data, we just add their logs. It's just a process of hill-climbing.
The top of that hill (log-likelihood as a function of $\theta$) has a sharpness, or curvature, which is its second derivative, and it's negative. You can calculate this numerically by moving $\theta$ around. The sharper it is, the more informative $\theta$ is.
If you turn the problem upside down so you are trying to minimize negative log-likelihood, it is the same problem, but now you are dealing in units of Shannon's information ( -log(P) ), so you could say you are minimizing the information in the observations.
Then when you take the second derivative, it is positive, and that's the Fisher Information.
Then all you have to do is extend $\theta$ to a vector, in which the second derivative becomes a matrix, the Fisher Information Matrix (FIM). If you maximize its determinant, that's called D-optimality.
To get the expected FIM, you assume ideal observations, and make assumptions about the distribution of the observational noise at each observation, and go from there. Even though the observations are in a sense "perfect", the log-likelihood function will still have a curvature.