# Why Generalized Least Squares?

So it is often advise to use Generalized Least Squares when we have a regression model with non-spherical(i.e. heteroskedastic or autocorrelated) errors. We do so by doing a weighted regression $$(y-x\hat\beta)^TW(y-x\hat\beta)$$ with $$W = \Sigma^{-1} = Cov(\epsilon)^{-1}$$, the inverse of the covariance matrix of errors.

The variance of the estimated $$\hat\beta$$ is \begin{aligned} Var(\hat\beta_{GLS})&=(X^TWX)^{-1}X^TW\Sigma W^TX(X^TW^TX)^{-1}\\ &=(X^TWX)^{-1}=(X^T\Sigma^{-1} W)^{-1} \end{aligned}

To do GLS, we must know $$\Sigma$$. But if we already know $$\Sigma$$, why can't we just do regular OLS, and calculate $$Var(\hat\beta)$$ as $$Var(\hat\beta)=(X^TX)^{-1}X^T\Sigma X(X^TX)$$ ? Is it because $$Var(\hat\beta_{GLS})$$ is smaller?

Another question I've always had is that for OLS, $$\beta$$ is estimated as : $$\hat\beta_{OLS}=(X^TX)^{-1}X^Ty$$ . For GLS or WLS, the $$\hat\beta$$ is estimated as $$\hat\beta_{GLS} = (X^TWX)^{-1}X^TWy$$ , which is unbiased for non-spherical error. Yet, we are told that $$\hat\beta_{OLS}$$ is also an unbiased estimator of $$\beta$$, even with non-spherical errors. Does that mean $$(X^TWX)^{-1}X^TWy$$ simplifies to $$(X^TX)^{-1}X^Ty$$?

You can indeed do regular OLS and compute the variance of the estimator which will be unbiased.

But GLS will be a more efficient estimator that has a lower variance of the sampling distribution (in fact out of all unbiased linear estimators, it will be the estimator with the least possible variance).

Example

Let us estimate $$\mu$$ with the following variables $$X_k \sim N\left(\mu, \sigma^2 \cdot k\right)$$

Then

$$\begin{array}{} \hat\mu_\text{OLS}& =&\frac{1}{n} \sum_{k=1}^n {X_k} &\sim& N\left(\mu, \sigma^2\cdot{\frac{1+1/n}{2}}\right)\\ \hat\mu_\text{GLS}& =&\frac{1}{H_{n,0.5}}\sum_{k=1}^n \frac{1}{\sqrt{k}} {X_k} &\sim& N\left(\mu, \sigma^2\cdot{\frac{n}{(H_{n,0.5})^2}}\right) \end{array}$$

where $$H_{n,0.5} = \sum_{k=1}^n \frac{1}{k^{0.5}}$$

The variance of the GLS estimator is smaller than the variance of the OLS estimator. So if you can reasonably guess the covariance matrix of the error distribution $$\Sigma$$ this may be beneficial. • That makes sense. What about the estimate of coefficient themselves? Is $\hat\beta_{OLS} = \hat\beta_{GLS}$? If not, how can both be unbiased estimators? Apr 16, 2022 at 22:16
• GLS estimates $\Sigma$. You just need to tell GLS the assumed correlation pattern (e.g., AR(1)) in order for it to estimate $\Sigma$. Apr 16, 2022 at 23:29
• @wwyws - if the estimates were always the same, neither would be better than the other. They can both be unbiased if both their expected values are the same (and equal to the true parameter); with each individual sample, though, their calculated values can still be different, as long as the differences have an expected value equal to zero. Apr 17, 2022 at 0:13
• @wwyws the example shows how the estimates are different. It is due to the different coefficients in the sum. But because the coefficients both sum up to 1, you get that the estimates have the expectation value $\mu$ and are unbiased. Apr 17, 2022 at 7:19