Why Generalized Least Squares?

Question

So it is often advise to use Generalized Least Squares when we have a regression model with non-spherical(i.e. heteroskedastic or autocorrelated) errors. We do so by doing a weighted regression $$ (y-x\hat\beta)^TW(y-x\hat\beta) $$ with $W = \Sigma^{-1} = Cov(\epsilon)^{-1}$, the inverse of the covariance matrix of errors.

The variance of the estimated $\hat\beta$ is $$ \begin{aligned} Var(\hat\beta_{GLS})&=(X^TWX)^{-1}X^TW\Sigma W^TX(X^TW^TX)^{-1}\\ &=(X^TWX)^{-1}=(X^T\Sigma^{-1} W)^{-1} \end{aligned} $$

To do GLS, we must know $\Sigma$. But if we already know $\Sigma$, why can't we just do regular OLS, and calculate $Var(\hat\beta)$ as $$ Var(\hat\beta)=(X^TX)^{-1}X^T\Sigma X(X^TX) $$ ? Is it because $Var(\hat\beta_{GLS})$ is smaller?

Another question I've always had is that for OLS, $\beta$ is estimated as : $$ \hat\beta_{OLS}=(X^TX)^{-1}X^Ty $$ . For GLS or WLS, the $\hat\beta$ is estimated as $$ \hat\beta_{GLS} = (X^TWX)^{-1}X^TWy $$ , which is unbiased for non-spherical error. Yet, we are told that $\hat\beta_{OLS}$ is also an unbiased estimator of $\beta$, even with non-spherical errors. Does that mean $(X^TWX)^{-1}X^TWy$ simplifies to $(X^TX)^{-1}X^Ty$?

Answer 1

You can indeed do regular OLS and compute the variance of the estimator (and that estimator will be unbiased and consistent).

But, GLS will be a more efficient estimator that has a lower variance of the sampling distribution (in fact out of all unbiased linear estimators, it will be the estimator with the least possible variance).

---

Example

Let us estimate $\mu$ with the following variables $$X_k \sim N\left(\mu, \sigma^2 \cdot k\right)$$

Then

$$\begin{array}{} \hat\mu_\text{OLS}& =&\frac{1}{n} \sum_{k=1}^n {X_k} &\sim& N\left(\mu, \sigma^2\cdot{\frac{1+1/n}{2}}\right)\\ \hat\mu_\text{GLS}& =&\frac{1}{H_{n,0.5}}\sum_{k=1}^n \frac{1}{\sqrt{k}} {X_k} &\sim& N\left(\mu, \sigma^2\cdot{\frac{n}{(H_{n,0.5})^2}}\right) \end{array}$$

where $H_{n,0.5} = \sum_{k=1}^n \frac{1}{k^{0.5}}$

The variance of the GLS estimator is smaller than the variance of the OLS estimator. So if you can reasonably guess the covariance matrix of the error distribution $\Sigma$ this may be beneficial.