How to do regression with known correlations among the errors?

Question

I was wondering if there is anything you can do when you have a regression problem:

$$\begin{cases} Y_t = \beta_1x_t + \beta_0 + \varepsilon_t \\ \left(\varepsilon_1,\ldots,\varepsilon_n\right)\sim\mathcal{N}_n\left(\mathbf{0},D \right) \\ 1\leq t \leq n \end{cases}$$

and you know $D$, the covariance matrix of the errors. I know weighted least squares when $D$ is diagonal. But what happens if it isn't? Is there any well established estimator for $(\beta_0,\beta_1)$ ?

Answer 1

In general, the Gauss Markov theorem gives that generalized least squares:

$$\hat{\beta}_{GLS} = \left( X^T D^{-1} X\right)^{-1} \left( X^T D^{-1} Y\right)$$

is the BLUE (best linear unbiased estimator) of $\beta$. See Seber and Lee (1973).

Note, even when $D$ is diagonal, it may not be proportional to $I$ by a factor of $\sigma^2$ (heteroscedatistic errors), so the resulting estimand gives the inverse variance weighted least squares for more efficient inference and estimation.

Answer 2

I had to write a proof that $\hat{\beta}_{GLS}$ is BLUE and the Seber and Lee reference recommended by @AdamO helped me a lot.

However I did not find an openly available version of the proof online. So I figured as I already did the work of typing it in LaTex, I might as well copy it here for those who have a hard time accessing Seber and Lee. The proof follows Seber and Lee closely. Please feel free to mention any typos or flaws in the argument.

Proof

• We would like to find a transform of the model that would lead us back to the homoskedastic case where we know that the simple version of Gauss-Markov applies (i.e. the Gauss-Markov theorem under homoskedasticity). Notice that because ${\rm Var}(e |X) = E(ee'|X) = D$, a diagonal matrix with positive diagonal entries, one can write $D = D^{1/2}D^{1/2}$, where $D^{1/2}$ is a diagonal matrix the diagonal elements of which are the square roots of the diagonal elements of $D$.

• Suppose $D$ is invertible, that is $\sigma_i >0$ for all $i = 1, \dots, n$. Then $D^{1/2}$ also has an inverse that we denote $D^{-1/2} $. So assume we have a model with errors $\gamma = D^{-1/2} e$ for some linear transformations of the $X$ and $y$ observations, say $Z$ and $w$. We would then have

\begin{align*} Var(\gamma |Z) & = E( \gamma \gamma'|Z) \\ &= E( D^{-1/2} e e' D^{-1/2} |Z) \\ &= D^{-1/2} E( e e' |Z) D^{-1/2} \\ &= D^{-1/2} D^{1/2}D^{1/2}D^{-1/2} \qquad ,\text{ as $Z$ is a linear transformation of $X$}\\ &= I \end{align*}

and this model would be homoskedastic.

• Notice that such configuration of the errors is provided by the model

  \begin{align*} \underbrace{D^{-1/2} y}_{:=w} = \underbrace{D^{-1/2}X}_{:= Z} \beta + \underbrace{D^{-1/2}e}_{:=\gamma} \end{align*}

• Also, applying the usual result for least-square estimation, we have that the value of $\beta$ which minimizes the squared errors in the transformed model is

  \begin{align*} \beta^* & = (Z'Z)^{-1} Z'w\\ &= ( X' D^{-1/2} D^{-1/2}X)^{-1} X' D^{-1/2} D^{-1/2} y \\ & = \underbrace{(X' D^{-1}X)^{-1} X' D^{-1}}_{:= A_0} y \\ \end{align*}

  the generalized least square estimator.

  • Because the transformed model is homoskedastic, $\beta^*$ is BLUE in the transformed model, by the simple Gauss-Markov theorem. So ${\rm Var}(\beta'|Z) - {\rm Var}(\beta^*|Z) $ is positive semi-definite for every $\beta'$, a linear unbiased estimation of the true parameter.

• Now notice that

  \begin{align*} & {\rm Var}(\beta^*|Z) = {\rm Var}(A_0' y|Z) = A_0' {\rm Var}(e|Z) A_0\quad\quad = A_0' D A_0\\ & {\rm Var}( \beta' | Z) = {\rm Var}(A' w|Z)\ = A' {\rm Var}(D^{-1/2}e|Z) A = A' D^{-1/2} D D^{-1/2} A = A'A \end{align*}

  and by the simple Gauss-Markov theorem we have $\tilde{A}'\tilde{A} - A_0' D A_0$ is positive semi-definite for any $\tilde{A}$ such that $\tilde{A}'Z = I$.

• Clearly, $A_0$ also yields an unbiased estimator in the original model, as $A_0' X = I$ (See Hansen, 4.6, freely available online).

• So to finally get back to the original heteroskedastic model, take any $A$ such that $A'X =I$ and let $\tilde{A}' = A'D^{-1/2}$.

• Notice that $\tilde{A}' Z = A'D^{-1/2} D^{1/2} X = A'X = I$. So by Gauss-Markov-first-part again, $\tilde{A}'\tilde{A} - A_0' D A_0$ is positive semi-definite.
• But $\tilde{A}'\tilde{A} = A'D^{-1/2}D^{1/2} A = A'DA$.
• So wrapping up we have $A_0 X = I$ and for any $A$ such that $AX =I$, $A'DA - A_0 DA_0$ is positive semi-definite, the desired result.

Answer 3

As a practical extension of @AdamO's response: Note that there are ways to find a "Square Root" of a matrix (the Cholesky decomposition is one example) such that $A'A=B$, so if you find such a root matrix of the $D^{-1}$ matrix mentioned above and multiply it by your $X$ matrix and $Y$ vector to get a new matrix and vector $X^* = AX$ and $Y^*=AY$ and plug $X^*$ and $Y^*$ into the regular OLS regression equation it is easy to derive the GLS formula given above. This means that if you have computer software that computes OLS regression, but not GLS regression you can just find a root matrix and multiply it against your data, then feed the transformed variables to the OLS routine and it will give you the estimates for the GLS fit.