Basic Question on Defining the Dimensions and Entries of the Fisher Information Matrix

Question

I'm a beginner trying to introduce myself to Maximum Likelihood Estimation (MLE) and can grasp some of the material adequately, such as looking for peaks in likelihood functions, approaching the Cramer-Rao Bound and inverting the Fisher Information Matrix to derive the covariance matrix and variance estimates. Ironically, I'm having more trouble trying to pin down precisely what the columns, rows and entries of the Fisher Matrix should represent and defining its structure. I have skimmed one or two dozen references over the past year or so (plus a search at CrossValidated) and have yet to see any examples with actual values plugged into the formulas, although their discussion of other aspects of the matrix is usually thorough.

I’ll try to explain in detail to avoid overlapping more advanced topics on other threads. Most of my confusion seems to stem from this: the integral in the matrix equation has a second-order derivative and likelihood function result on the divisor and two derivatives for a single parameter on the dividend, usually with two subscripts like i and j. (See this Wikipedia entry and this one for examples). There always seem to be exactly two subscripts, which implies a two-dimensional matrix and makes me ask the related questions below:

1. If I had only one treatment and one parameter, that seems to imply a one-dimensional matrix. If so, what would the matrix subscripts i and j refer to? Would I need to do a cross product of the single dimension to itself to derive the dividend in the Fisher Matrix equation?

2. How would the column and row structures change if I had two treatments and needed to estimate a single parameter? Would this imply a 2D matrix?

3. Would the converse situation, where there's one treatment and two parameters (say scale and shape) make a difference to Question #2? I imagine this wouldn't be practical for some distributions, if one of the parameters was needed to derive the other in the likelihood function.

4. How would I alter the matrix structure and calculate the entries if I have two or more treatments plus two or more parameters? This seems to imply a 3D or higher matrix, in which case we'd need more subscripts than just i and j. I have yet to see any formulas to that effect in the texts, journal articles and tutorials I've skimmed to date though (I have a list of references if necessary). Is this commonly done in real-world MLEs?

5. Can we expand the matrix to include separate distributions or even distribution families, along with their parameters? How would this affect the structure of the matrix?

6. Can the matrix entries consist of a mix of calculations on both likelihood and observed values, if the latter are available? The Fisher Metric formula at https://en.wikipedia.org/wiki/Fisher_information_metric#Definition seems to substitute PDFs for likelihoods. Would this constitute mixing observed information with Fisher Information? This part of the question may lead into other topics like the subtle differences between observed and Fisher info that are probably covered better elsewhere. I'm just wondering here if the two types of entries are ever mixed in the same matrix. I’m assuming they’d almost always be kept separate.

I realize that the answers I'm searching for are probably no-brainers; I'm obviously getting some simple underlying concept wrong. Once I get past this stumbling block, I should be able to quickly plug some probability functions into the Fisher formulas, return some covariance matrices and practice selecting some MLEs; ordinarily that would be the hard part, but I'm stuck on this basic task. A picture is worth a thousand words so to speak: the answers to above questions would probably be instantly clear, if I saw examples with actual values plugged in. All that would remain then is to explain how to populate the matrix from the usual formula using only two subscripts, or alternately, any changes to the formula to accommodate multiple treatments and parameters. Links to any such examples or exercises would also be helpful. Thanks in advance :)

Answer 1

The Fisher information is a symmetric square matrix with a number of rows/columns equal to the number of parameters you're estimating. Recall that it's a covariance matrix of the scores, & there's a score for each parameter; or the expectation of the negative of a Hessian, with a gradient for each parameter. When you want to consider different experimental treatments you represent their effects by adding more parameters to the model; i.e. more rows/columns (rather than more dimensions—a matrix has two dimensions by definition). When you're estimating only a single parameter, the Fisher information is just a one-by-one matrix (a scalar)—the variance of, or the expected value of the negative of the second derivative of, the score.

For a simple linear regression model of $Y$ on $x$ with $n$ observations

$y_i = \beta_0 +\beta_1 x_i + \varepsilon_i$

where $\varepsilon \sim \mathrm{N}(0,\sigma^2)$, there are three parameters to estimate, the intercept $\beta_0$, the slope $\beta_1$, & the error variance $\sigma^2$ ; the Fisher information is

$$ \begin{align} \mathcal{I}(\beta_0,\beta_1,\sigma^2) =& \operatorname{E} \left[ \begin{matrix} \left(\tfrac{\partial \ell(\beta_0,\beta_1,\sigma^2)}{\partial \beta_0}\right)^2 & \tfrac{\partial \ell(\beta_0,\beta_1,\sigma^2)}{\partial \beta_0} \tfrac{\partial \ell(\beta_0,\beta_1,\sigma^2)}{\partial \beta_1} & \tfrac{\partial \ell(\beta_0,\beta_1,\sigma^2)}{\partial \beta_0}\tfrac{\partial \ell(\beta_0,\beta_1,\sigma^2)}{\partial \sigma^2}\\ \tfrac{\partial \ell(\beta_0,\beta_1,\sigma^2)}{\partial \beta_1}\tfrac{\partial \ell(\beta_0,\beta_1,\sigma^2)}{\partial \beta_0} & \left(\tfrac{\partial \ell(\beta_0,\beta_1,\sigma^2)}{\partial \beta_1}\right)^2& \tfrac{\partial \ell(\beta_0,\beta_1,\sigma^2)}{\partial \beta_1}\tfrac{\partial \ell(\beta_0,\beta_1,\sigma^2)}{\partial \sigma^2}\\ \tfrac{\partial \ell(\beta_0,\beta_1,\sigma^2)}{\partial \sigma^2}\tfrac{\partial \ell(\beta_0,\beta_1,\sigma^2)}{\partial \beta_0} & \tfrac{\partial \ell(\beta_0,\beta_1,\sigma^2)}{\partial \sigma^2}\tfrac{\partial \ell(\beta_0,\beta_1,\sigma^2)}{\partial \beta_1} & \left(\tfrac{\partial \ell(\beta_0,\beta_1,\sigma^2)}{\partial \sigma^2}\right)^2\\ \end{matrix} \right] \\ \\ =& -\operatorname{E}\left[ \begin{matrix} \tfrac{\partial^2 \ell(\beta_0,\beta_1,\sigma^2)}{(\partial \beta_0)^2} & \tfrac{\partial^2 \ell(\beta_0,\beta_1,\sigma^2)}{\partial \beta_0 \partial \beta_1} & \tfrac{\partial^2 \ell(\beta_0,\beta_1,\sigma^2)}{\partial \beta_0\partial \sigma^2}\\ \tfrac{\partial^2 \ell(\beta_0,\beta_1,\sigma^2)}{\partial \beta_1\partial \beta_0} & \tfrac{\partial^2 \ell(\beta_0,\beta_1,\sigma^2)}{(\partial \beta_1)^2} & \tfrac{\partial^2 \ell(\beta_0,\beta_1,\sigma^2)}{\partial \beta_1\partial \sigma^2}\\ \tfrac{\partial^2 \ell(\beta_0,\beta_1,\sigma^2)}{\partial \sigma^2\partial \beta_0} & \tfrac{\partial^2 \ell(\beta_0,\beta_1,\sigma^2)}{\partial \sigma^2\partial \beta_1} & \tfrac{\partial^2 \ell(\beta_0,\beta_1,\sigma^2)}{(\partial \sigma^2)^2}\\ \end{matrix} \right]\\ \\ =& \left[ \begin{matrix} \tfrac{n}{\sigma^2} & \tfrac{\sum_i^n x_i}{\sigma^2} & 0\\ \tfrac{\sum_i^n x_i}{\sigma^2} & \tfrac{\sum_i^n x_i^2}{\sigma^2} & 0\\ 0 & 0 & \tfrac{n}{2\sigma^4} \end{matrix} \right] \end{align} $$

where $\ell(\cdot)$ is the log-likelihood function of the parameters. (Note that $x$ might be a dummy variable indicating a particular treatment.)