4
$\begingroup$

I was wondering why R uses the Iterated Re-weighted Least Square in the GLM case?

When defining the problem analytically using Newton-Raphson / Fisher scoring, one comes to the following (vector notation):

$$\beta_{t+1} = \beta_t + (X^TWX)^{-1}X^TWM(y-\mu)$$

Then a "trick" is employed, to transform this to:

$$\beta_{t+1} = (X^TWX)^{-1}X^TW(X\beta_t + M(y-\mu)) := (X^TWX)^{-1}X^TWZ_t$$

And the solution is found using the Re-Weighted Least Squares algorithm.

I wonder why this extra trick/step is needed? Is it just because the algorithm used to solve RWLS is more efficient and numerically stable (uses QR decomposition)? Or is there some other underlying reason?

Improve this question
$\endgroup$
7
  • 1
    $\begingroup$ I don't know the answer but john fox's book "compantion to applied regression" discusses the two approaches so the answer might be in there. $\endgroup$
    – mlofton
    Commented Feb 9, 2021 at 16:30
  • $\begingroup$ @mlofton he barely mentions the topic. $\endgroup$
    – Maverick Meerkat
    Commented Feb 9, 2021 at 19:33
  • $\begingroup$ Hi: are you looking at the companion or the actual text. He went into it pretty deeply in the old version of CAR. Maybe he took it out of the updated version ? It definitely was in there back in say 2010. $\endgroup$
    – mlofton
    Commented Feb 10, 2021 at 15:49
  • $\begingroup$ I'm looking into "An R Companion to Applied Regression" if that's what you meant... I can't find the answer there. Also looked at Generalized Linear Models 2nd edition by McCullagh and Nelder. But they don't explain either why they use IRLS... They just show that it's equivalent to Fisher-Scoring. My best guess so far is that there's already optimized code to solve Weighted LS, so they rather use that instead of writing new code just for this GLM Fisher-Scoring case, but I'm not sure. $\endgroup$
    – Maverick Meerkat
    Commented Feb 10, 2021 at 16:04
  • $\begingroup$ I did mean that. It could be an edition issue but maybe my recall ability of what book discusses what is failing !!!!! ( I'm getting old ). See if this helps at all and apologies that I led you in wrong direction. stats.stackexchange.com/questions/205/… $\endgroup$
    – mlofton
    Commented Feb 10, 2021 at 16:52

1 Answer 1

Reset to default
2
$\begingroup$

In the book I'm reading now (Generalized Estimating Equations by James W. Hardin , and Joseph M. Hilbe) they explain (p. 7):

"[In the early 1970's] There was a clear need to find an optimization method by which otherwise nonlinear models could be estimated using standard OLS methods. Wedderburn and Nelder discovered that the methods used to estimate weighted linear regression could be adjusted to model many data situations that were previously estimated via maximum likelihood, particularly for those maximum likelihood models based on the exponential family of distributions. They accomplished this by applying the Iterative Weighted Least Squares (IWLS) algorithm already in use. In addition, they employed a link function which linearized such functions as the logistic, probit, and log."

I.e., they wanted to use existing computer programs to model GLM's. My guess is that today IWLS/IRLS is not needed, but lingers on as "legacy" code/math-derivation.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.