Covariance of linear regression coefficients in weighted least squares method

Question

I'm a Physics student and we had a lab where we had to process some data in the form $x_i$, $y_i$, $\sigma_i$, where the $\sigma_i$'s are the uncertainties of the $y_i$'s.

With the weighted least squares method we obtained coefficients $A$ and $B$, the estimated parameters for a straight line in the form $y = A+Bx$ that would fit the data. In particular, $A=(s_x s_{xx} - s_x s_{xy})/D$, $B=(s_1 s_{xy} - s_x s_y)/D$, where

$D=s_1 s_{xx} - s_x s_x$,

$s_1 = \sum_i \frac{1}{\sigma_i^2}$, $s_{xx} = \sum_i \frac{x_i^2}{\sigma_i^2}$, $s_{xy} = \sum_i \frac{x_i y_i}{\sigma_i^2}$, $s_x = \sum_i \frac{x_i}{\sigma_i^2}$, $s_y = \sum_i \frac{y_i}{\sigma_i^2}$.

In addition we have named $\delta_{AB}$ the variable $-s_x/D$. In light of the use of $\delta_{AB}$ in the subsequent data analysis, it seems that this variable represents the covariance between $A$ and $B$.

In my statistics courses we never covered the covariance between linear regression coefficients, so my question is what is the general theory behind $\delta_{AB}$? Is it possible to prove simply, in the simple case of two variables, that this is indeed the covariance between $A$ and $B$?

Answer 1

The general (multiple regression) case is actually easier than trying to do simple regression.

$y = X\beta + \epsilon$ where $\operatorname{Var}(\epsilon) = \operatorname{diag}(\sigma_i^2) = \sigma^2 \operatorname{diag}(\tau_i^2) $

Let $W^{-1} = \operatorname{diag}(\tau_i^2)$. Assume $X$ is of full rank.

$\hat{\beta} = \left(X'WX\right)^{-1} X'Wy$

---

Edit2:

A couple of rules related to expectation and variance, for some random vector $z$:

$\operatorname{E}(Az) = A\operatorname{E}(z)$

$\operatorname{Var}(Az) = A\operatorname{Var}(x)A'$

You can confirm this reasonably easily by expanding out both sides and looking at it term by term (keeping in mind that the (i,j) off-diagonal term of $\operatorname{Var}(z)$ is $\operatorname{Cov}(z_i,z_j)\,$).

Edit:

Where $\hat{\beta}$ comes from:

As before, $y = X\beta + \epsilon$ is the underlying (population) equation.

Let $y = X\hat{\beta} + e$ be the fitted equation.

Let $S = \sum_{i=1}^n w_i e_i^2 = e'We$, the sum of weighted squared residuals.

Note that

$S = e'We = (y - X\hat{\beta})'W(y - X\hat{\beta})$

$=y'Wy-y'WX\hat{\beta}-\hat{\beta}'X'Wy+\hat{\beta}'X'WX\hat{\beta}$

$=y'Wy-2\hat{\beta}'X'Wy+\hat{\beta}'X'WX\hat{\beta}$

We wish to find the $\hat{\beta}$ which will minimise $S$ (making the fit go as close as possible to the data in the WLS sense).

So we do that by calculus.

\begin{align} \frac{\partial S}{\partial \hat{\beta}} &= \frac{\partial (y'Wy-2\hat{\beta}'X'Wy+\hat{\beta}'X'WX\hat{\beta})}{\partial \hat{\beta}} \\ &= (0-2X'Wy+2X'WX\hat{\beta}) \end{align}

(Note that the second derivative will be $X'WX$, which is positive definite, so it's a minimum.)

To find a turning point, we set that first derivative equal to $0$ and solve for $\hat{\beta}$:

\begin{align} -2X'Wy+2X'WX\hat{\beta} &= 0\\ X'WX\hat{\beta}&=X'Wy\\ \hat{\beta}&=(X'WX)^{-1}X'Wy\\ \end{align}

---

\begin{align} \operatorname{Var}(\hat{\beta}) &= \operatorname{Var}((X'WX)^{-1}X'Wy) \\ &=\left(X'WX\right)^{-1} X'W \operatorname{Var}(y)\, WX\left(X'WX\right)^{-1}\\ &= \sigma^2 \left(X'WX\right)^{-1} X' W X\left(X'WX\right)^{-1}\\ & = \sigma^2 \left(X'WX\right)^{-1} \end{align}

With $\sigma^2$ estimated, this gives the usual estimate of the variance-covariance matrix of parameters.

Now in your case

$$ X'WX = \begin{bmatrix} \sum w_i & \sum w_i x_i\\ \sum w_i x_i & \sum w_i x_i^2 \end{bmatrix} $$

So

$$ \left(X'WX\right)^{-1} = \frac{1}{\Delta} \begin{bmatrix} \sum w_i x_i^2 & -\sum w_i x_i\\ -\sum w_i x_i & \sum w_i \end{bmatrix} $$

where $\Delta=\sum w_i \sum w_i x_i^2 - \sum w_i x_i \sum w_i x_i$

So $\operatorname{Cov}(\hat{\beta_0},\hat{\beta_1}) = -\frac{\sigma^2 \sum w_i x_i}{\Delta}$

Which matches your formula. That is, looks like you got it right, $\delta_{AB}$ is the covariance of the parameters for a weighted LS line.

However, your approach relies on the $\sigma_i$ values being exactly right; mine only requires they be proportional to right (given we can estimate $\sigma^2$).

When you say "what is the general theory behind $\delta_{AB}$?" I have to admit I'm not sure what you're asking for - do you want its distribution or something? Or just a derivation something like the one given?