I am trying to answer the following question:

Given $\vec{\epsilon} \sim N(0,\Sigma)$ vector where $\epsilon_i$ are NOT iid. Find the estimator $\hat{\beta}$ that minimises $(Y − X\beta)^T\Sigma^{−1}(Y − X\beta)$. Find the mean squared error of this estimator.

My approach:

$(Y − X\beta)^T\Sigma^{−1}(Y − X\beta)$ is $N(0,I_n)$ in distribution. Where $\beta$ is the usual ordinary least squares estimator.

In that case, the estimator does NOT change in case of heteroscedasticity, does it?

If $X^T$ is full rank, then

$\hat{\beta}=(X^TX)^{-1}X^TY$,from where $\hat{\beta} \sim N(\beta, (X^TX)^{-1}X^T\Sigma X(X^TX)^{-1})$

And the bias of $\hat{\beta}$ is $0$. So I find:

$$\begin{eqnarray} \text{Quadratic risk} &=& \text{Variance} + \text{bias}^2 \\ &=& (X^TX)^{-1}X^T\Sigma X(X^TX)^{-1} \\ \end{eqnarray}$$ Is this right?

  • $\begingroup$ The OLS does not minimize the mean squared error when data are heteroscedastic. See the Gauss-Markov theorem. $\endgroup$ – AdamO May 15 '18 at 17:15
  • $\begingroup$ Multiply out the function you want to minimize and then find the $\hat{\beta}$ which minimizes it by taking the first derivative, setting it equal to 0, and solving. It's pretty straightforward. $\endgroup$ – klumbard May 15 '18 at 17:29

In case of heteroskedasticity the estimator actually changes:

If $Var$($\varepsilon$$\mid$$X$) = $\Sigma$ $\ne$ $\sigma^{2}$$I_{n}$, despite heteorskedasticity, we still have that $\Sigma$ is positive defined $n$ $\times$ $n$ matrix.

This implies that $\Sigma$ can be factorized as follows:

$\Sigma$ = $C$$\Lambda$$C'$

where $\Lambda$ is an $($$n$$\times$$n$$)$ diagonal matrix with the main diagonal elements all strictly positive and $C$ is an $n$$\times$$n$ matrix such that $C$$C'$ = $C'$$C$ = $I$.

Such decompostion of $\Sigma$ is useful since it canbe proved that:

$\Sigma^{-1}$ = $C$$\Lambda^{-1}$$C'$


$\Sigma^{-1/2}$ = $C$$\Lambda^{-1/2}$$C'$.

The Genarlized Least Squares transformed model is defined as:

$\Lambda^{-1/2}$$C'$$y$ = $\Lambda^{-1/2}$$C'$$X$$\beta$ + $\Lambda^{-1/2}$$C'$$\varepsilon$

from which we finally find the $\hat{\beta_{Gls}}$ applying to it the OLS:

$\hat{\beta_{Gls}}$ = $($$X'$$C$$\Lambda^{-1/2}$$\Lambda^{-1/2}$$C'$$X$$)^{-1}$$X'$$C$$\Lambda^{-1/2}$$\Lambda^{-1/2}$$C'$$y$ = $($$X'$$\Sigma^{-1}$$X$$)^{-1}$$X'$$\Sigma^{-1}$$y$


$Var$$($$\hat{\beta_{Gls}}$$\mid$$X$$)$ = $($$X'$$\Sigma^{-1}$$X$$)^{-1}$.

Note that sometimes, for heteroskedastic $\varepsilon$, the Variance matrix is further decomposed as follows:

$\Sigma$ = $\sigma^{2}$$\Omega$


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.