I am trying to understand the derivation of the MLE estimates of $\beta=(\beta_1,\dots,\beta_p)$ in Generalized Linear Models. The elements of the Information matrix are given by: $$ J_{jk}=\sum_{i=1}^N \frac{ x_{ij}x_{ik}}{Var(Y_i)}\left(\frac{\partial \mu_i}{\partial \eta_i}\right)^2 $$ and the derivation of this makes sense to me. But then we have the decomposition: $$ J=X^TWX $$ Where the diagonal elements of $W$ are : $$ W_{ii}=\frac{1}{Var(Y_i)}\left(\frac{\partial \mu_i}{\partial \eta_i}\right)^2 $$ My matrix algebra is a little off, and I'm having trouble visualising this decomposition and finding the non-diagonal elements of $W$. $J$ is a $p \times p$ matrix, and so it must be that $X$ is given by: $$ \begin{pmatrix} x_{11} & x_{12}&\cdots & x_{1p} \\ x_{21} & x_{22}&\cdots & x_{2p} \\ \vdots & \vdots& \ddots & \vdots \\ x_{N1} & x_{N2}&\cdots & x_{Np} \\ \end{pmatrix} $$ and let's say $W$ is given by: $$ \begin{pmatrix} w_{11} & w_{12}&\cdots & w_{1N} \\ w_{21} & w_{22}&\cdots & w_{2N} \\ \vdots & \ddots & \vdots \\ w_{N1} & w_{N2}&\cdots & w_{NN} \\ \end{pmatrix} $$

If we take the first part and define: $M = X^TW$, then we have that the elements of $M$ are given by: $$ M_{jk} = \sum_{i=1}^N x_{ji}w_{ik} $$ and finally, the elements of $J$ are given by: $$ J_{jk} = \sum_{z=1}^NM_{jz}x_{zk} = \sum_{z=1}^N \left(\sum_{i=1}^N x_{ji} w_{iz} \right)x_{zk} $$ after expanding this sum and comparing it to the definition of $J_{jk}$ im still having trouble understanding what the elements of $W$ actually are, is there an easier way to think about this?

  • $\begingroup$ Your $X$-matrix is wrong. Look at the last column. Note also that $W$ should be diagonal. $\endgroup$
    – Glen_b
    Mar 26, 2016 at 1:31
  • $\begingroup$ @Glen_b that was a typo for W, it should be an $N \times N$ matrix. From the set up of the GLM problem, we have $Y_i$ iid from the exponential family, with $g(E(Y_i)) = g(\mu_i)= x_i^T \beta$ where $x_i = (X_{i1},X_{i2},\cdots, X_{ip})$, which is how I got $X$, I'm not quite sure why this is wrong? $\endgroup$ Mar 26, 2016 at 1:37
  • $\begingroup$ @Glen_b woops, sorry, I see what you meant now $\endgroup$ Mar 26, 2016 at 1:39

1 Answer 1


Note that you have $W$ fully populated (entries for all $n^2$ elements). As I said in comments, $W$ should be diagonal (all off-diagonal elements are 0); you specified what the diagonal elements were at the start of your question. That is

$W_{ii}=\frac{1}{Var(Y_i)}\left(\frac{\partial \mu_i}{\partial \eta_i}\right)^2$

All other elements are 0.

By way of example, imagine a Poisson model with log-link. $\text{Var}(Y_i)=\mu_i$ and $\frac{\partial \mu_i}{\partial \eta_i}=e^{\eta_i}=\mu_i$. As a result, $W_{ii}=\mu_i$ and $W_{ij}=0$ for $j\neq i$.

  • $\begingroup$ ah, I think I over complicated my thinking about it, missed the fact that it said diagonal matrix $\endgroup$ Mar 26, 2016 at 1:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy