# Information decomposition in GLM MLE derivation

I am trying to understand the derivation of the MLE estimates of $\beta=(\beta_1,\dots,\beta_p)$ in Generalized Linear Models. The elements of the Information matrix are given by: $$J_{jk}=\sum_{i=1}^N \frac{ x_{ij}x_{ik}}{Var(Y_i)}\left(\frac{\partial \mu_i}{\partial \eta_i}\right)^2$$ and the derivation of this makes sense to me. But then we have the decomposition: $$J=X^TWX$$ Where the diagonal elements of $W$ are : $$W_{ii}=\frac{1}{Var(Y_i)}\left(\frac{\partial \mu_i}{\partial \eta_i}\right)^2$$ My matrix algebra is a little off, and I'm having trouble visualising this decomposition and finding the non-diagonal elements of $W$. $J$ is a $p \times p$ matrix, and so it must be that $X$ is given by: $$\begin{pmatrix} x_{11} & x_{12}&\cdots & x_{1p} \\ x_{21} & x_{22}&\cdots & x_{2p} \\ \vdots & \vdots& \ddots & \vdots \\ x_{N1} & x_{N2}&\cdots & x_{Np} \\ \end{pmatrix}$$ and let's say $W$ is given by: $$\begin{pmatrix} w_{11} & w_{12}&\cdots & w_{1N} \\ w_{21} & w_{22}&\cdots & w_{2N} \\ \vdots & \ddots & \vdots \\ w_{N1} & w_{N2}&\cdots & w_{NN} \\ \end{pmatrix}$$

If we take the first part and define: $M = X^TW$, then we have that the elements of $M$ are given by: $$M_{jk} = \sum_{i=1}^N x_{ji}w_{ik}$$ and finally, the elements of $J$ are given by: $$J_{jk} = \sum_{z=1}^NM_{jz}x_{zk} = \sum_{z=1}^N \left(\sum_{i=1}^N x_{ji} w_{iz} \right)x_{zk}$$ after expanding this sum and comparing it to the definition of $J_{jk}$ im still having trouble understanding what the elements of $W$ actually are, is there an easier way to think about this?

• Your $X$-matrix is wrong. Look at the last column. Note also that $W$ should be diagonal. Mar 26, 2016 at 1:31
• @Glen_b that was a typo for W, it should be an $N \times N$ matrix. From the set up of the GLM problem, we have $Y_i$ iid from the exponential family, with $g(E(Y_i)) = g(\mu_i)= x_i^T \beta$ where $x_i = (X_{i1},X_{i2},\cdots, X_{ip})$, which is how I got $X$, I'm not quite sure why this is wrong? Mar 26, 2016 at 1:37
• @Glen_b woops, sorry, I see what you meant now Mar 26, 2016 at 1:39

Note that you have $W$ fully populated (entries for all $n^2$ elements). As I said in comments, $W$ should be diagonal (all off-diagonal elements are 0); you specified what the diagonal elements were at the start of your question. That is
$W_{ii}=\frac{1}{Var(Y_i)}\left(\frac{\partial \mu_i}{\partial \eta_i}\right)^2$
By way of example, imagine a Poisson model with log-link. $\text{Var}(Y_i)=\mu_i$ and $\frac{\partial \mu_i}{\partial \eta_i}=e^{\eta_i}=\mu_i$. As a result, $W_{ii}=\mu_i$ and $W_{ij}=0$ for $j\neq i$.