2
$\begingroup$

A question on a problem sheet I am working on asks us to find the Fisher information matrix (FIM) for the following setup:
Suppose that $Y_i \sim \textrm{Bin}(r_i, \pi_i)$ for $i = 1, 2, \dots , n$, all independent, where the $r_i$ are known, $\ln({\pi_i/(1 − \pi_i)}) = \beta_0 + \beta_1 x_i$ and $x_i$ is a known covariate. We would like to find the FIM.
When finding the FIM for a model using the identity link, it is as simple as finding derivatives of the log-likelihood, here it seems more challenging. At first I wrote $\pi_i =\frac{\exp(\beta_0 + \beta_1 x_i)}{1+\exp(\beta_0 + \beta_1 x_i)}$ and attempted to differentiate the log-likelihood with respect to the parameters, but that seems to be incorrect. The solution sheet says that the following solves the problem:
"We know that $\mu_i = \pi_i r_i$ and $\eta_i=\ln(\pi_i/(1 − \pi_i)) =\ln(\mu_i /(r_i - \mu_i))$ So that $$\frac{d \eta_i}{d \mu_i}= \frac{r_i}{\mu_i(r_i- \mu_i)} = \frac{1}{r_i \pi_i(1-\pi_i)} $$Since $\operatorname{Var}(Y_i)=r_i \pi_i(1-\pi_i)$ the Fisher information matrix is given by:

$$V= \begin{pmatrix}\sum_{i=1}^nr_i \pi_i(1-\pi_i) & \sum_{i=1}^nx_ir_i \pi_i(1-\pi_i)\\ \sum_{i=1}^nx_ir_i \pi_i(1-\pi_i) & \sum_{i=1}^nx_i^2r_i \pi_i(1-\pi_i) \end{pmatrix}.$$

I am very unsure how this solution was achieved; can anyone break this down a bit more?

$\endgroup$
5
  • $\begingroup$ You seem to be confusing information and variance for one point. The asymptotic GLM result is that $\sqrt{n} \left( \hat{\beta} - \beta \right) \rightarrow_d \mathcal{N}(0, \mathcal{I}^{-1}(\beta))$ $\endgroup$
    – AdamO
    Dec 9, 2022 at 17:55
  • $\begingroup$ @AdamO The only reference to variance for one point in the question is in the quoted solution from the solution sheet. The whole point of the question is to compute $\cal I$ and the formula for $\cal I$ in standard GLM theory involves the variance function, so the variance of an individual point is relevant. $\endgroup$ Dec 10, 2022 at 21:05
  • $\begingroup$ @GordonSmyth I don't understand your point. The question clearly has inconsistencies referring to variance ("V") and calling it information. $\endgroup$
    – AdamO
    Dec 11, 2022 at 0:22
  • $\begingroup$ @AdamO The V matrix is inside quotes so, if it was wrong, then it would be fault of the solution sheet rather than the fault of OP. However the V matrix shown in the question is in fact the correct information matrix for this logistic regression. The reason why the entries look like variances is because the logit link derivatives in this case are equal to inverse variances. $\endgroup$ Dec 11, 2022 at 1:02
  • $\begingroup$ @AdamO It is also worth clarifying that $\cal I$ in your comment represents average information per observation, i.e., it is the usual information matrix divided by $n$. The information matrix that is used in the glm fit, and which is being derived by OP, is not divided by $n$. $\endgroup$ Dec 11, 2022 at 2:46

1 Answer 1

3
$\begingroup$

The model setup is a binomial generalized linear model with logit link, also called logistic regression. There are standard and quite simple formulas for the Fisher information matrix (FIM) of a generalized linear model. You do not need to differentiate the log-likelihood from the scratch because the log-likelihood for a GLM has a special form that simplifies the calculations considerably. Indeed you only need to know the variance var$(y)$ and the link-derivative $\partial\mu/\partial \eta$ mentioned in the solution sheet, then the FIM can be written down immediately. The only difference between the logit link and the identity link is that the $\partial\mu/\partial \eta$ term changes.

You are working through a problem sheet, which is presumably associated with a lecture series or online course on generalized linear models and statistical inference. The course you are studying will presumably have derived formula for the FIM of a generalized linear model, and the purpose of the problem sheet is to prompt you to use the formulas and methods that have been derived in the course. So my advice is to go back to the course that the problem sheet is from and familiarize yourself with the methods that have been presented in the course. Otherwise, working through the problem sheet won't have any learning value for you.

If you want to consult a textbook, we derive the same information matrix as in your question in Equation 4.16 of my book with Peter Dunn (Dunn & Smyth, 2018). We go through the calculations in quite a lot of detail. Our example assumes $r_i=1$ but that makes little difference to the calculations.

Reference

Dunn PK, Smyth GK (2018). Generalized linear models with examples in R. Springer, New York, NY. DOI: 10.1007/978-1-4419-0118-7. ISBN: 978-1-4419-0118-7.

$\endgroup$
1
  • 1
    $\begingroup$ I think this is a good piece of advice along with proper reference which I deem enough for OP to flex their muscle to reach to the conclusion. +1. $\endgroup$ Dec 11, 2022 at 2:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.