2
$\begingroup$

The algorithm, used to optimize the likelihood function in a generalized linear model (GLM) such as poisson regression, is iterated weighted least square (IWLS), where the Newton-Raphson algorithm is used.

The single Newton-Raphson update is: $$ \begin{aligned} \beta^{new} &= \beta^{old} - \left(\frac{\partial^{2} L}{\partial \beta \partial \beta^{T}}\right)^{-1} \frac{\partial L}{\partial \beta} \\ &= \beta^{old} + (X^{T}WX)^{-1}X^{T}(V), \end{aligned} $$ where $\frac{\partial^{2} L}{\partial \beta \partial \beta^{T}}$ = $-X^{T}WX$, and W is a diagonal matrix whose elements are, for example, usually functions of inverse link function in GLM, and the derivatives are evaluated at $\beta^{old}$.

My question is that in my case, I cannot ensure that all the elements in W are positive, sometimes even negative values occurred, which causes positive definite problem. What should I do now?

Improve this question
$\endgroup$
1
  • 1
    $\begingroup$ Can you give more detail on why you can't ensure all weights are positive? $\endgroup$
    – jld
    Commented Mar 2, 2017 at 15:50

1 Answer 1

Reset to default
2
$\begingroup$

If I'm understanding your question right, you should be able to fix this by using ridge regression. As I show here, doing IRLS with an $L_2$ penalty on the likelihood turns into doing IRLS with $X^T W X + \lambda I$ in place of $X^T W X$, i.e. each step of IRLS is a weighted ridge regression. For $\lambda$ sufficiently large, $X^T W X + \lambda I$ will be diagonally dominant and therefore PD.

Improve this answer
$\endgroup$
5
  • $\begingroup$ Thanks for the prompt response, the direction, and the link, it is helpful $\endgroup$
    – vtshen
    Commented Mar 2, 2017 at 22:22
  • $\begingroup$ Besides Ridge method, I was curious if there is any other idea/method/algorithm to solve this problem? $\endgroup$
    – vtshen
    Commented Nov 9, 2018 at 22:29
  • $\begingroup$ @vtshen would you be able to add more details as to where the negative weights are coming from? In general if you can’t ensure that an optimization is convex then you might just have to settle for a local optimum. But I’d be surprised if that’s necessary for an actual GLM $\endgroup$
    – jld
    Commented Nov 16, 2018 at 12:47
  • 1
    $\begingroup$ @ jld Thanks so much for the continuous help on this problem. It is not very easy for me to describe in detail that why the negative weights are introduced in my problem. But basically, this GLM problem is not a classical GLM. I added some modifications to the GLM loss function, and that change introduced the negative weights. However, I agree that finding the global optimum is too optimistic, I think it is more reasonable to search for methods dealing with local optimum $\endgroup$
    – vtshen
    Commented Nov 17, 2018 at 0:20
  • $\begingroup$ @vtshen are you finding that the matrix $X^TWX$ is becoming singular, or rather it's just not positive definite? You could explore fixing the singularity by adding a small bit to the diagonal like $X^TWX + 10^{-10}I$ just to condition it a bit better although if it has negative eigenvalues this wouldn't necessarily guarantee invertibility, or maybe use a pseudoinverse. But if the issue is that you have both positive and negative eigenvalues, that just means you've got a nonconvex problem and you might just have to accept that. You could still happily get a local optimum hopefully $\endgroup$
    – jld
    Commented Nov 20, 2018 at 18:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.