# The quadratic risk of beta estimate in heteroscedastic regression

I am trying to answer the following question:

Given $\vec{\epsilon} \sim N(0,\Sigma)$ vector where $\epsilon_i$ are NOT iid. Find the estimator $\hat{\beta}$ that minimises $(Y − X\beta)^T\Sigma^{−1}(Y − X\beta)$. Find the mean squared error of this estimator.

My approach:

$(Y − X\beta)^T\Sigma^{−1}(Y − X\beta)$ is $N(0,I_n)$ in distribution. Where $\beta$ is the usual ordinary least squares estimator.

In that case, the estimator does NOT change in case of heteroscedasticity, does it?

If $X^T$ is full rank, then

$\hat{\beta}=(X^TX)^{-1}X^TY$,from where $\hat{\beta} \sim N(\beta, (X^TX)^{-1}X^T\Sigma X(X^TX)^{-1})$

And the bias of $\hat{\beta}$ is $0$. So I find:

$$\begin{eqnarray} \text{Quadratic risk} &=& \text{Variance} + \text{bias}^2 \\ &=& (X^TX)^{-1}X^T\Sigma X(X^TX)^{-1} \\ \end{eqnarray}$$ Is this right?

• The OLS does not minimize the mean squared error when data are heteroscedastic. See the Gauss-Markov theorem. – AdamO May 15 '18 at 17:15
• Multiply out the function you want to minimize and then find the $\hat{\beta}$ which minimizes it by taking the first derivative, setting it equal to 0, and solving. It's pretty straightforward. – klumbard May 15 '18 at 17:29

## 1 Answer

In case of heteroskedasticity the estimator actually changes:

If $Var$($\varepsilon$$\mid$$X$) = $\Sigma$ $\ne$ $\sigma^{2}$$I_{n}, despite heteorskedasticity, we still have that \Sigma is positive defined n \times n matrix. This implies that \Sigma can be factorized as follows: \Sigma = C$$\Lambda$$C' where \Lambda is an ($$n$$\times$$n$$) diagonal matrix with the main diagonal elements all strictly positive and C is an n$$\times$$n matrix such that C$$C'$ = $C'$$C = I. Such decompostion of \Sigma is useful since it canbe proved that: \Sigma^{-1} = C$$\Lambda^{-1}$$C' and: \Sigma^{-1/2} = C$$\Lambda^{-1/2}$$C'. The Genarlized Least Squares transformed model is defined as: \Lambda^{-1/2}$$C'$$y = \Lambda^{-1/2}$$C'$$X$$\beta$ + $\Lambda^{-1/2}$$C'$$\varepsilon$

from which we finally find the $\hat{\beta_{Gls}}$ applying to it the OLS:

$\hat{\beta_{Gls}}$ = $($$X'$$C$$\Lambda^{-1/2}$$\Lambda^{-1/2}$$C'$$X$$)^{-1}$$X'$$C$$\Lambda^{-1/2}$$\Lambda^{-1/2}$$C'$$y = ($$X'$$\Sigma^{-1}$$X$$)^{-1}$$X'$$\Sigma^{-1}$$y$

with:

$Var$$($$\hat{\beta_{Gls}}$$\mid$$X$$) = ($$X'$$\Sigma^{-1}$$X$$)^{-1}. Note that sometimes, for heteroskedastic \varepsilon, the Variance matrix is further decomposed as follows: \Sigma = \sigma^{2}$$\Omega$