# Negative diagonal elements in the weight matrix of the iterated weighted least square (IWLS) in GLM

The algorithm, used to optimize the likelihood function in a generalized linear model (GLM) such as poisson regression, is iterated weighted least square (IWLS), where the Newton-Raphson algorithm is used.

The single Newton-Raphson update is: \begin{aligned} \beta^{new} &= \beta^{old} - (\frac{\partial^{2} L}{\partial \beta \partial \beta^{T}})^{-1} \frac{\partial L}{\partial \beta} \\ &= \beta^{old} + (X^{T}WX)^{-1}X^{T}(V), \end{aligned} where $\frac{\partial^{2} L}{\partial \beta \partial \beta^{T}}$ = $-X^{T}WX$, and W is a diagonal matrix whose elements are, for example, usually functions of inverse link function in GLM, and the derivatives are evaluated at $\beta^{old}$.

My question is that in my case, I cannot ensure that all the elements in W are positive, sometimes even negative values occurred, which causes positive definite problem. What should I do now?

• Can you give more detail on why you can't ensure all weights are positive?
– jld
Mar 2, 2017 at 15:50

If I'm understanding your question right, you should be able to fix this by using ridge regression. As I show here, doing IRLS with an $L_2$ penalty on the likelihood turns into doing IRLS with $X^T W X + \lambda I$ in place of $X^T W X$, i.e. each step of IRLS is a weighted ridge regression. For $\lambda$ sufficiently large, $X^T W X + \lambda I$ will be diagonally dominant and therefore PD.
• @vtshen are you finding that the matrix $X^TWX$ is becoming singular, or rather it's just not positive definite? You could explore fixing the singularity by adding a small bit to the diagonal like $X^TWX + 10^{-10}I$ just to condition it a bit better although if it has negative eigenvalues this wouldn't necessarily guarantee invertibility, or maybe use a pseudoinverse. But if the issue is that you have both positive and negative eigenvalues, that just means you've got a nonconvex problem and you might just have to accept that. You could still happily get a local optimum hopefully