3
$\begingroup$

I'm a beginner trying to introduce myself to Maximum Likelihood Estimation (MLE) and can grasp some of the material adequately, such as looking for peaks in likelihood functions, approaching the Cramer-Rao Bound and inverting the Fisher Information Matrix to derive the covariance matrix and variance estimates. Ironically, I'm having more trouble trying to pin down precisely what the columns, rows and entries of the Fisher Matrix should represent and defining its structure. I have skimmed one or two dozen references over the past year or so (plus a search at CrossValidated) and have yet to see any examples with actual values plugged into the formulas, although their discussion of other aspects of the matrix is usually thorough.

I’ll try to explain in detail to avoid overlapping more advanced topics on other threads. Most of my confusion seems to stem from this: the integral in the matrix equation has a second-order derivative and likelihood function result on the divisor and two derivatives for a single parameter on the dividend, usually with two subscripts like i and j. (See this Wikipedia entry and this one for examples). There always seem to be exactly two subscripts, which implies a two-dimensional matrix and makes me ask the related questions below:

  1. If I had only one treatment and one parameter, that seems to imply a one-dimensional matrix. If so, what would the matrix subscripts i and j refer to? Would I need to do a cross product of the single dimension to itself to derive the dividend in the Fisher Matrix equation?

  2. How would the column and row structures change if I had two treatments and needed to estimate a single parameter? Would this imply a 2D matrix?

  3. Would the converse situation, where there's one treatment and two parameters (say scale and shape) make a difference to Question #2? I imagine this wouldn't be practical for some distributions, if one of the parameters was needed to derive the other in the likelihood function.

  4. How would I alter the matrix structure and calculate the entries if I have two or more treatments plus two or more parameters? This seems to imply a 3D or higher matrix, in which case we'd need more subscripts than just i and j. I have yet to see any formulas to that effect in the texts, journal articles and tutorials I've skimmed to date though (I have a list of references if necessary). Is this commonly done in real-world MLEs?

  5. Can we expand the matrix to include separate distributions or even distribution families, along with their parameters? How would this affect the structure of the matrix?

  6. Can the matrix entries consist of a mix of calculations on both likelihood and observed values, if the latter are available? The Fisher Metric formula at https://en.wikipedia.org/wiki/Fisher_information_metric#Definition seems to substitute PDFs for likelihoods. Would this constitute mixing observed information with Fisher Information? This part of the question may lead into other topics like the subtle differences between observed and Fisher info that are probably covered better elsewhere. I'm just wondering here if the two types of entries are ever mixed in the same matrix. I’m assuming they’d almost always be kept separate.

I realize that the answers I'm searching for are probably no-brainers; I'm obviously getting some simple underlying concept wrong. Once I get past this stumbling block, I should be able to quickly plug some probability functions into the Fisher formulas, return some covariance matrices and practice selecting some MLEs; ordinarily that would be the hard part, but I'm stuck on this basic task. A picture is worth a thousand words so to speak: the answers to above questions would probably be instantly clear, if I saw examples with actual values plugged in. All that would remain then is to explain how to populate the matrix from the usual formula using only two subscripts, or alternately, any changes to the formula to accommodate multiple treatments and parameters. Links to any such examples or exercises would also be helpful. Thanks in advance :)

$\endgroup$
  • 2
    $\begingroup$ +1 because you really to have tried to answer it yourself but as a friendly advice: Go ahead and edit this into a smaller question. To that extent, can you not numerically evaluate a log-likelihood cost function, gets its Hessian and see for yourself how the partial derivatives look like? (I can do that for you if you want.) Start with a continuous variable first not a discrete. (I think you also confuse what more parameters do; they only add columns to your design matrix, not full dimensions.) $\endgroup$ – usεr11852 Feb 20 '16 at 5:16
  • $\begingroup$ Thanks for the advice - I'll try editing it a few times tonight and tomorrow to chop it down. I'd be interested in seeing the process for deriving the matrix with one parameters vs. two vs. multiple parameters and treatments. That's the real sticking point for me; I can't envision how to handle these three scenarios differently, given the version of the Fisher formula with two subscripts. Thanks :) $\endgroup$ – SQLServerSteve Feb 20 '16 at 6:46
  • $\begingroup$ The most succinct way to answer this would be screenshots/links of actual Fisher matrices with 1) one parameter 2) two parameters & 3) multiple parameters with multiple treatments, as long as the quantities in each column/row are clearly labeled. Or a simple, "With multiple parameters & treatments, we would have X # of columns and Y # of Rows.With one parameter, the values would go here; X. etc" Formulas/walkthroughs are unnecessary, unless changes are needed to accommodate extra parameters & treatments. I can & ought to do that myself; I just need finished examples to compare my structure to. $\endgroup$ – SQLServerSteve Feb 21 '16 at 1:37
  • $\begingroup$ This looks rather daunting to answer. I think it might be easier to give an example & show where you're stuck in the derivation or confused about the interpretation. But, for a start, what's a "treatment" mean? An observation? An experimental treatment? $\endgroup$ – Scortchi Feb 21 '16 at 16:49
  • $\begingroup$ @Scortchi The answer's probably unusually simple. The use of exactly 2 subscripts i & j in the formula is the only point where I'm stuck in the derivation; I can't grasp how that can accommodate single parameters, or multiple parameters/treatments. Viewing actual finished Fisher Information Matrices with clearly labeled columns & rows would answer it; there just aren't many in the literature. I'd then say, "Aha, this is why 2 subscripts can handle 3 parameters or just 1, etc.Here is where they would placed in the matrix." It's just the structure of the finished product I'm after, nothing more. $\endgroup$ – SQLServerSteve Feb 21 '16 at 21:52
4
$\begingroup$

The Fisher information is a symmetric square matrix with a number of rows/columns equal to the number of parameters you're estimating. Recall that it's a covariance matrix of the scores, & there's a score for each parameter; or the expectation of the negative of a Hessian, with a gradient for each parameter. When you want to consider different experimental treatments you represent their effects by adding more parameters to the model; i.e. more rows/columns (rather than more dimensions—a matrix has two dimensions by definition). When you're estimating only a single parameter, the Fisher information is just a one-by-one matrix (a scalar)—the variance of, or the expected value of the negative of the second derivative of, the score.

For a simple linear regression model of $Y$ on $x$ with $n$ observations

$y_i = \beta_0 +\beta_1 x_i + \varepsilon_i$

where $\varepsilon \sim \mathrm{N}(0,\sigma^2)$, there are three parameters to estimate, the intercept $\beta_0$, the slope $\beta_1$, & the error variance $\sigma^2$ ; the Fisher information is

$$ \begin{align} \mathcal{I}(\beta_0,\beta_1,\sigma^2) =& \operatorname{E} \left[ \begin{matrix} \left(\tfrac{\partial \ell(\beta_0,\beta_1,\sigma^2)}{\partial \beta_0}\right)^2 & \tfrac{\partial \ell(\beta_0,\beta_1,\sigma^2)}{\partial \beta_0} \tfrac{\partial \ell(\beta_0,\beta_1,\sigma^2)}{\partial \beta_1} & \tfrac{\partial \ell(\beta_0,\beta_1,\sigma^2)}{\partial \beta_0}\tfrac{\partial \ell(\beta_0,\beta_1,\sigma^2)}{\partial \sigma^2}\\ \tfrac{\partial \ell(\beta_0,\beta_1,\sigma^2)}{\partial \beta_1}\tfrac{\partial \ell(\beta_0,\beta_1,\sigma^2)}{\partial \beta_0} & \left(\tfrac{\partial \ell(\beta_0,\beta_1,\sigma^2)}{\partial \beta_1}\right)^2& \tfrac{\partial \ell(\beta_0,\beta_1,\sigma^2)}{\partial \beta_1}\tfrac{\partial \ell(\beta_0,\beta_1,\sigma^2)}{\partial \sigma^2}\\ \tfrac{\partial \ell(\beta_0,\beta_1,\sigma^2)}{\partial \sigma^2}\tfrac{\partial \ell(\beta_0,\beta_1,\sigma^2)}{\partial \beta_0} & \tfrac{\partial \ell(\beta_0,\beta_1,\sigma^2)}{\partial \sigma^2}\tfrac{\partial \ell(\beta_0,\beta_1,\sigma^2)}{\partial \beta_1} & \left(\tfrac{\partial \ell(\beta_0,\beta_1,\sigma^2)}{\partial \sigma^2}\right)^2\\ \end{matrix} \right] \\ \\ =& -\operatorname{E}\left[ \begin{matrix} \tfrac{\partial^2 \ell(\beta_0,\beta_1,\sigma^2)}{(\partial \beta_0)^2} & \tfrac{\partial^2 \ell(\beta_0,\beta_1,\sigma^2)}{\partial \beta_0 \partial \beta_1} & \tfrac{\partial^2 \ell(\beta_0,\beta_1,\sigma^2)}{\partial \beta_0\partial \sigma^2}\\ \tfrac{\partial^2 \ell(\beta_0,\beta_1,\sigma^2)}{\partial \beta_1\partial \beta_0} & \tfrac{\partial^2 \ell(\beta_0,\beta_1,\sigma^2)}{(\partial \beta_1)^2} & \tfrac{\partial^2 \ell(\beta_0,\beta_1,\sigma^2)}{\partial \beta_1\partial \sigma^2}\\ \tfrac{\partial^2 \ell(\beta_0,\beta_1,\sigma^2)}{\partial \sigma^2\partial \beta_0} & \tfrac{\partial^2 \ell(\beta_0,\beta_1,\sigma^2)}{\partial \sigma^2\partial \beta_1} & \tfrac{\partial^2 \ell(\beta_0,\beta_1,\sigma^2)}{(\partial \sigma^2)^2}\\ \end{matrix} \right]\\ \\ =& \left[ \begin{matrix} \tfrac{n}{\sigma^2} & \tfrac{\sum_i^n x_i}{\sigma^2} & 0\\ \tfrac{\sum_i^n x_i}{\sigma^2} & \tfrac{\sum_i^n x_i^2}{\sigma^2} & 0\\ 0 & 0 & \tfrac{n}{2\sigma^4} \end{matrix} \right] \end{align} $$

where $\ell(\cdot)$ is the log-likelihood function of the parameters. (Note that $x$ might be a dummy variable indicating a particular treatment.)

$\endgroup$
  • $\begingroup$ Perfect - it's exactly what I needed. I'll mull this overnight and see if I need any clarifications - I can't spot any right now - but this answer already addresses all of the varied scenarios I mentioned above, in one fell swoop. Thanks $\endgroup$ – SQLServerSteve Feb 22 '16 at 19:54
  • $\begingroup$ The structure of @Scortchi 's example clearly shows how the Fisher formula I linked to needs only two matrix subscripts - i and j - to accommodate any number of parameters and values. Each non-diagonal in the top matrix has exactly two terms in the dividend; instead of adding or subtracting terms from each dividend, each unique combination of parameters adds or subtracts rows and columns from the matrix. Most published literature doesn't make that important distinction clear, which led to my confusion. $\endgroup$ – SQLServerSteve Feb 22 '16 at 21:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.