I am trying to understand the derivation of the MLE estimates of $\beta=(\beta_1,\dots,\beta_p)$ in Generalized Linear Models. The elements of the Information matrix are given by: $$ J_{jk}=\sum_{i=1}^N \frac{ x_{ij}x_{ik}}{Var(Y_i)}\left(\frac{\partial \mu_i}{\partial \eta_i}\right)^2 $$ and the derivation of this makes sense to me. But then we have the decomposition: $$ J=X^TWX $$ Where the diagonal elements of $W$ are : $$ W_{ii}=\frac{1}{Var(Y_i)}\left(\frac{\partial \mu_i}{\partial \eta_i}\right)^2 $$ My matrix algebra is a little off, and I'm having trouble visualising this decomposition and finding the non-diagonal elements of $W$. $J$ is a $p \times p$ matrix, and so it must be that $X$ is given by: $$ \begin{pmatrix} x_{11} & x_{12}&\cdots & x_{1p} \\ x_{21} & x_{22}&\cdots & x_{2p} \\ \vdots & \vdots& \ddots & \vdots \\ x_{N1} & x_{N2}&\cdots & x_{Np} \\ \end{pmatrix} $$ and let's say $W$ is given by: $$ \begin{pmatrix} w_{11} & w_{12}&\cdots & w_{1N} \\ w_{21} & w_{22}&\cdots & w_{2N} \\ \vdots & \ddots & \vdots \\ w_{N1} & w_{N2}&\cdots & w_{NN} \\ \end{pmatrix} $$

If we take the first part and define: $M = X^TW$, then we have that the elements of $M$ are given by: $$ M_{jk} = \sum_{i=1}^N x_{ji}w_{ik} $$ and finally, the elements of $J$ are given by: $$ J_{jk} = \sum_{z=1}^NM_{jz}x_{zk} = \sum_{z=1}^N \left(\sum_{i=1}^N x_{ji} w_{iz} \right)x_{zk} $$ after expanding this sum and comparing it to the definition of $J_{jk}$ im still having trouble understanding what the elements of $W$ actually are, is there an easier way to think about this?

  • $\begingroup$ Your $X$-matrix is wrong. Look at the last column. Note also that $W$ should be diagonal. $\endgroup$
    – Glen_b
    Mar 26, 2016 at 1:31
  • $\begingroup$ @Glen_b that was a typo for W, it should be an $N \times N$ matrix. From the set up of the GLM problem, we have $Y_i$ iid from the exponential family, with $g(E(Y_i)) = g(\mu_i)= x_i^T \beta$ where $x_i = (X_{i1},X_{i2},\cdots, X_{ip})$, which is how I got $X$, I'm not quite sure why this is wrong? $\endgroup$ Mar 26, 2016 at 1:37
  • $\begingroup$ @Glen_b woops, sorry, I see what you meant now $\endgroup$ Mar 26, 2016 at 1:39

1 Answer 1


Note that you have $W$ fully populated (entries for all $n^2$ elements). As I said in comments, $W$ should be diagonal (all off-diagonal elements are 0); you specified what the diagonal elements were at the start of your question. That is

$W_{ii}=\frac{1}{Var(Y_i)}\left(\frac{\partial \mu_i}{\partial \eta_i}\right)^2$

All other elements are 0.

By way of example, imagine a Poisson model with log-link. $\text{Var}(Y_i)=\mu_i$ and $\frac{\partial \mu_i}{\partial \eta_i}=e^{\eta_i}=\mu_i$. As a result, $W_{ii}=\mu_i$ and $W_{ij}=0$ for $j\neq i$.

  • $\begingroup$ ah, I think I over complicated my thinking about it, missed the fact that it said diagonal matrix $\endgroup$ Mar 26, 2016 at 1:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.