# Best OLS estimators

Hi i am stuck on this one, the question is related to Gauss-Markov theorem:

Consider a general alternative to the OLS estimator that is also a linear unbiased estimator, say ${\tilde \beta}$. Outline a proof that the OLS estimator b is better, in a well defined sense, than ${\tilde \beta}$.

For both estimators b and ${\tilde \beta}$ i have to use the fact that $E(UU') = \sigma^2I$.

I have shown already that $b= \left(\mathbf {X}'\mathbf {X}\right)^{-1}\mathbf {X}'\mathbf {y} = \left(\mathbf {X'}\mathbf {X}\right)^{-1}\mathbf X'(\mathbf {X}\mathbf {\beta}+u)$

$=\left(\mathbf {X'}\mathbf {X}\right)^{-1}\mathbf X'\mathbf {X}\mathbf \beta+(X'X)^{-1}X'u$

$=\beta+(X'X)^{-1}X'u$

Theorem proved for linearity, now

$E(b)= E[ \beta+(X'X)^{-1}X'u]=\beta$ as $E[u]=0$

Theorem proved for Unbiasedness.

Variance-covariance matrix of b: $Var(b)=E(b-\beta)(b-\beta)'$ using again $E(UU') = \sigma^2I$

$=E(b-\beta)(b-\beta)'=\sigma^2(X'X)^{-1}$

in order to show that b is a better estimator that ${\tilde \beta}$ i need to follow the same reasoning still using the fact that $E(UU') = \sigma^2I$ and then conclude that b has the minimum variance of all estimators thus it is the best, that is

$Var(b)\leq Var(\tilde \beta)$ i guess.

I am not sure how to properly compare the two estimators so if someone could give me a hint it would be much appreciated.

• The estimators will be considered in terms of their variance-covariance matrices and the difference (competitor's (-) ols') should be a positive semi-definite matrix. You have to find a way to express the competitor estimator in connection to OLS (hint: both are linear functions of the data (y). Commented Jan 6, 2015 at 18:42
• thanks Alecos, i actually found it using $\tilde \beta \equiv (A+C)y = ((X'X)^{-1}X'+ C)y$ Commented Jan 6, 2015 at 18:53
• Exactly. But please post the solution as an answer here, so that the thread becomes useful to other people also. Commented Jan 6, 2015 at 19:07
• If you're 'stuck on a question', this should probably be tagged self-study. See the tag wiki Commented Feb 26, 2015 at 20:55

We can compute the variance-covariance matrice of b and $\tilde \beta$ and hence compare their variances in order to tell which one has the smallest variance. (Best estimator)

Knowing tha $E(U)=0$

$E(b-\beta)(b-\beta)'= E(\beta+(X'X)^{-1}X'u-\beta)(\beta+(X'X)^{-1}X'u-\beta)'$

$=E[(X'X)^{-1}X'u][(X'X)^{-1}X'u]$

$=E[(X'X)^{-1}X'uu'X(X'X)^{-1})$ recall that $E(uu')= \sigma^2I$

$=\sigma^2(X'X)^{-1}X'IX(X'X)^{-1}$

$=\sigma^2(X'X)^{-1}$

So it has been shown that b is unbiased as $E(b_i)=\beta_i$ and therefore $E(b)=\beta$ but also linear as $b=(X'X)^{-1}X'y$, $b=Ay$ with $A=(X'X)^{-1}X'$

Let $\tilde \beta$ a general linear unbiased estimator, $\tilde \beta\equiv(A+C)y=[(X'X)^{-1}X'+C)y$

$=[(X'X)^{-1}X'+C)(X\beta+u)$

$=\beta+(X'X)^{-1}X'u+CX\beta+Cu$, we require $CX=0$ $\forall$ $X$ for unbiasedness

$E(\tilde \beta)=E[\beta+(X'X)^{-1}Xu + CX\beta+Cu]$ with $CX=0$ required and $E(u)=0$

$=E(\beta)$

Hence $\tilde \beta$ is also unbiased and $\tilde \beta=b+Cy$ is linear

Variance-covariance matrix of $\tilde \beta$: $E(\tilde \beta-\beta)(\tilde \beta-\beta)'=E[(X'X)^{-1}X'u+Cu)[(X'X)^{-1}Xu+Cu)']$

$=E[(X'X)^{-1}X'uu'X(X'X)^{-1}+(X'X)^{-1}X'uu'C']+Cuu'X(X'X)^{-1}+Cuu'C'$

using the fact that $E(uu')=\sigma^2I$ which simplifies to

$E(\tilde \beta-\beta)(\tilde \beta-\beta)'= \sigma^2(X'X)^{-1}+\sigma^2(X'X)^{-1}X'C'+\sigma^2CX(X'X)^{-1}+\sigma^2CC'$

we know that $CX=0$ hence $(CX)'=X'C'=0$ so we have

$E(\tilde \beta-\beta)(\tilde \beta-\beta)'=\sigma^2(X'X)^{-1}+\sigma^2CC'$

$=\sigma^2[(X'X)^{-1}+CC']$ , Recall that $E(b-\beta)(b -\beta)'=\sigma^2(X'X)^{-1}$

Thus $E(\tilde \beta-\beta)(\tilde \beta-\beta)'=E(b-\beta)(b -\beta)'+ \sigma^2CC'$

We can conclude that $Var(b)\leq Var(\tilde \beta)$ which means that b is the best estimator.

• (+1) Thanks for contributing. I would just add for clarity the obvious, that the result holds because $CC'$ is positive semidefinite matrix. Commented Jan 7, 2015 at 1:54
• I had to solve the same problem at University lately and found your answer difficult to go through. Here you will find a simpler approach leading to the same result. economictheoryblog.com/2015/02/26/markov_theorem Commented Feb 26, 2015 at 16:29