"General" normal equations?

Question

I am quite familiar with the usual OLS estimate for $\boldsymbol\beta$, given by $$\hat{\boldsymbol\beta} = (X^{T}X)^{-1}X^{T}\mathbf{Y}$$ for the linear model $\mathbf{Y} = X\boldsymbol\beta + \boldsymbol\epsilon$.

My professor presented that instead of assuming $\text{Var}[\mathbf{Y}] = \sigma^2I$ ($I$ being the identity matrix), if $\text{Var}[\mathbf{Y}] = \Sigma$, then $$\hat{\boldsymbol\beta} = (X^{T}\Sigma^{-1}X)^{-1}X^{T}\Sigma^{-1}\mathbf{Y}\text{.}$$

This was presented without proof. How is this derived? I don't see where in particular the original derivation is dependent on $\text{Var}[\mathbf{Y}] = \sigma^2I$.

Answer 1

We can always (attempt to) solve the optimization problem $$\hat \beta = \textrm{argmin}_{\beta \in \mathbb R^p} ||Y - X\beta||^2$$

but this solution doesn't necessarily have the properties that we want if it is not the case that $Y = X\beta + \varepsilon$ where $E(\varepsilon_i) = 0$, $Var(\varepsilon_i) = \sigma^2$, and $Cov(\varepsilon_i, \varepsilon_j) = 0$ for $i \neq j$. For instance, if $Var(\varepsilon) \neq \sigma^2 I_n$ then we don't have the Gauss-Markov theorem.

If we instead do

$$ \hat \beta = \textrm{argmin}_{\beta \in \mathbb R^p} (Y - X\beta)^T \Sigma^{-1} (Y - X\beta) $$ (which is minimizing the Mahalanobis distance) then we continue to enjoy properties that we want. This is called generalized least squares and results in the estimator that you saw.

As the wikipedia article mentions, you can also view this as OLS on a transformation of your data where you multiply through by $\Sigma^{-1/2}$ (assuming $\Sigma$ is positive definite).