Covariance of linear regression coefficients in weighted least squares method

I'm a Physics student and we had a lab where we had to process some data in the form $x_i$, $y_i$, $\sigma_i$, where the $\sigma_i$'s are the uncertainties of the $y_i$'s.

With the weighted least squares method we obtained coefficients $A$ and $B$, the estimated parameters for a straight line in the form $y = A+Bx$ that would fit the data. In particular, $A=(s_x s_{xx} - s_x s_{xy})/D$, $B=(s_1 s_{xy} - s_x s_y)/D$, where

$D=s_1 s_{xx} - s_x s_x$,

$s_1 = \sum_i \frac{1}{\sigma_i^2}$, $s_{xx} = \sum_i \frac{x_i^2}{\sigma_i^2}$, $s_{xy} = \sum_i \frac{x_i y_i}{\sigma_i^2}$, $s_x = \sum_i \frac{x_i}{\sigma_i^2}$, $s_y = \sum_i \frac{y_i}{\sigma_i^2}$.

In addition we have named $\delta_{AB}$ the variable $-s_x/D$. In light of the use of $\delta_{AB}$ in the subsequent data analysis, it seems that this variable represents the covariance between $A$ and $B$.

In my statistics courses we never covered the covariance between linear regression coefficients, so my question is what is the general theory behind $\delta_{AB}$? Is it possible to prove simply, in the simple case of two variables, that this is indeed the covariance between $A$ and $B$?

• While @Glen_b gave you an excellent answer, I would like to note that if your $\sigma_i^2$ (or the proportionality constants $\tau_i$, in his notation) are not very accurate, then you may end up being worse off with your regression than you would have assuming equal variances. Econometricians have given this topic a lot of thought in the 1970s-1980s. Mar 20, 2013 at 15:30

The general (multiple regression) case is actually easier than trying to do simple regression.

$$y = X\beta + \epsilon$$ where $$\operatorname{Var}(\epsilon) = \operatorname{diag}(\sigma_i^2) = \sigma^2 \operatorname{diag}(\tau_i^2)$$

Let $$W^{-1} = \operatorname{diag}(\tau_i^2)$$. Assume $$X$$ is of full rank.

$$\hat{\beta} = \left(X'WX\right)^{-1} X'Wy$$

Edit2:

A couple of rules related to expectation and variance, for some random vector $$z$$:

$$\operatorname{E}(Az) = A\operatorname{E}(z)$$

$$\operatorname{Var}(Az) = A\operatorname{Var}(x)A'$$

You can confirm this reasonably easily by expanding out both sides and looking at it term by term (keeping in mind that the (i,j) off-diagonal term of $$\operatorname{Var}(z)$$ is $$\operatorname{Cov}(z_i,z_j)\,$$).

Edit:

Where $$\hat{\beta}$$ comes from:

As before, $$y = X\beta + \epsilon$$ is the underlying (population) equation.

Let $$y = X\hat{\beta} + e$$ be the fitted equation.

Let $$S = \sum_{i=1}^n w_i e_i^2 = e'We$$, the sum of weighted squared residuals.

Note that

$$S = e'We = (y - X\hat{\beta})'W(y - X\hat{\beta})$$

$$=y'Wy-y'WX\hat{\beta}-\hat{\beta}'X'Wy+\hat{\beta}'X'WX\hat{\beta}$$

$$=y'Wy-2\hat{\beta}'X'Wy+\hat{\beta}'X'WX\hat{\beta}$$

We wish to find the $$\hat{\beta}$$ which will minimise $$S$$ (making the fit go as close as possible to the data in the WLS sense).

So we do that by calculus.

\begin{align} \frac{\partial S}{\partial \hat{\beta}} &= \frac{\partial (y'Wy-2\hat{\beta}'X'Wy+\hat{\beta}'X'WX\hat{\beta})}{\partial \hat{\beta}} \\ &= (0-2X'Wy+2X'WX\hat{\beta}) \end{align}

(Note that the second derivative will be $$X'WX$$, which is positive definite, so it's a minimum.)

To find a turning point, we set that first derivative equal to $$0$$ and solve for $$\hat{\beta}$$:

\begin{align} -2X'Wy+2X'WX\hat{\beta} &= 0\\ X'WX\hat{\beta}&=X'Wy\\ \hat{\beta}&=(X'WX)^{-1}X'Wy\\ \end{align}

\begin{align} \operatorname{Var}(\hat{\beta}) &= \operatorname{Var}((X'WX)^{-1}X'Wy) \\ &=\left(X'WX\right)^{-1} X'W \operatorname{Var}(y)\, WX\left(X'WX\right)^{-1}\\ &= \sigma^2 \left(X'WX\right)^{-1} X' W X\left(X'WX\right)^{-1}\\ & = \sigma^2 \left(X'WX\right)^{-1} \end{align}

With $$\sigma^2$$ estimated, this gives the usual estimate of the variance-covariance matrix of parameters.

$$X'WX = \begin{bmatrix} \sum w_i & \sum w_i x_i\\ \sum w_i x_i & \sum w_i x_i^2 \end{bmatrix}$$

So

$$\left(X'WX\right)^{-1} = \frac{1}{\Delta} \begin{bmatrix} \sum w_i x_i^2 & -\sum w_i x_i\\ -\sum w_i x_i & \sum w_i \end{bmatrix}$$

where $$\Delta=\sum w_i \sum w_i x_i^2 - \sum w_i x_i \sum w_i x_i$$

So $$\operatorname{Cov}(\hat{\beta_0},\hat{\beta_1}) = -\frac{\sigma^2 \sum w_i x_i}{\Delta}$$

Which matches your formula. That is, looks like you got it right, $$\delta_{AB}$$ is the covariance of the parameters for a weighted LS line.

However, your approach relies on the $$\sigma_i$$ values being exactly right; mine only requires they be proportional to right (given we can estimate $$\sigma^2$$).

When you say "what is the general theory behind $$\delta_{AB}$$?" I have to admit I'm not sure what you're asking for - do you want its distribution or something? Or just a derivation something like the one given?

• Thank you! Your answer is already very useful for me. About the general theory my question is basically where I can find a theorethical explanation of your derivation, i.e. as you said, what is the distribution of the $\beta$'s, and so on. The problem is that in none of my books there is a good theoretical foundation for the method of weighted least squares. As an aside, could you please elaborate on what are $\sigma^2$ and the $\tau_i$'s in your derivation? Mar 20, 2013 at 9:54
• The τ's basically correspond to your σ's (though possibly scaled). I did this simply because it makes the results more closely match the usual weighted regression derivation. If, for example, there was additional variation over your σ's, then one possible model would be to add a scaling factor on the variance ($σ^2$ in my formulation). If you don't want that, just let my $σ^2=1$, and replace my $\tau$s with σ. Mar 20, 2013 at 10:07
• When you say 'a theoretical explanation' of my derivation, what is it you would like? I can do how $\hat{\beta}$ was derived. Would you like more than that? Would you be happier with it purely done in simple regression terms? Mar 20, 2013 at 10:09
• Yes, how $\hat{\beta}$ and $Var(\hat{\beta})$ were derived. But I don't want to waste your time, just a good reference would suffice Mar 20, 2013 at 10:11
• Sorry I didn't see your reply until I had finished doing those two (while doing the first it occurred to me you'd need the second as well). I have not explained it in great detail, so feel free to ask for where particular steps come about. Mar 20, 2013 at 10:44