Return to Question

When performing weighted least squares $L = \frac{1}{2} \sum_i w_i r_i^2$, Aitken showed that one ought to weight each sample by the inverse of its variance $w_i=1/\sigma_i^2$. This leads to gradients of the form

$$\nabla_\beta L = \sum_i \frac{r_i}{\sigma_i^2}\nabla_\beta r_i $$

In GLMs, the log-likelihood is $\ell(\beta) = \sum_i \frac{\theta_i y_i - b(\theta_i)}{\phi_i} + c(y_i, \phi_i)\qquad$ (cf. Turners notes)

for which the gradients are $\frac{\partial \ell}{\partial \beta} =\sum_i \frac{y_{i}-\mu_{i}}{\operatorname{V}[\mu_i]} \cdot \frac{x_{i}}{g^{\prime}\left(\mu_{i}\right)} $. This looks suspiciously similar to the WLS gradient. What is the precise connections between the two? Superficially it seems like all a GLM is really doing is  (1) transforming the prediction into a useful domain via the link function  (2) weighing the gradient updates 'optimally' under the assumption that $\sigma_i^2=V[\mu_i$]

When performing weighted least squares $L = \frac{1}{2} \sum_i w_i r_i^2$, Aitken showed that one ought to weight each sample by the inverse of its variance $w_i=1/\sigma_i^2$. This leads to gradients of the form

$$\nabla_\beta L = \sum_i \frac{r_i}{\sigma_i^2}\nabla_\beta r_i $$

In GLMs, the log-likelihood is $\ell(\beta) = \sum_i \frac{\theta_i y_i - b(\theta_i)}{\phi_i} + c(y_i, \phi_i)\qquad$ (cf. Turner's notes)

for which the gradients are $\frac{\partial \ell}{\partial \beta} =\sum_i \frac{y_{i}-\mu_{i}}{\operatorname{V}[\mu_i]} \cdot \frac{x_{i}}{g^{\prime}\left(\mu_{i}\right)} $. 

This looks suspiciously similar to the WLS gradient. What are the precise connections between the two? Superficially it seems like all a GLM is really doing is 

(1) transforming the prediction into a useful domain via the link function 

(2) weighting the gradient updates 'optimally' under the assumption that $\sigma_i^2=V[\mu_i]$.