I was learning the differences between least squares and weighted least squares in terms of estimating $\beta$ for $$ y = x\beta + \epsilon $$ where $y$ and $x$ are $n\times 1$ vector and $\epsilon$ is not i.i.d, but $\epsilon \sim N(0, \sigma^2\Omega)$, where $\Omega$ is a $n \times n$ nonsingular, positive definite and symmetric matrix, assumed to be known.

I was learning from these linear regression analysis slides, with a vanilla linear regression (assuming $\epsilon$ i.i.d), we can estimate $\beta$ as $$ \hat{\beta}_1 = (x^Tx)^{-1}x^Ty $$ with the expectation and variance as: $$ E[\hat{\beta}_1] = (x^Tx)^{-1}x^TE[y] = \beta $$ $$ V(\hat{\beta}_1) = (x^Tx)^{-1}x^TV(y)x(x^Tx)^{-1} = \sigma^2(x^Tx)^{-1}x^T\Omega x(x^Tx)^{-1} $$ Then, with weighted least squares, we can estimate $\beta$ as $$ \hat{\beta}_2 = (x^T\Omega^{-1}x)^{-1}x^T\Omega^{-1}y $$ with expectation and variance as: $$ E[\hat{\beta}_2] = \beta $$ $$ V(\hat{\beta}_2) = \sigma^2(x^T\Omega^{-1}x)^{-1} $$ and the author says $\hat{\beta}_2$ is the best linear unbiased estimator of $\beta$

I have these following questions:

  1. How to define the best linear unbiased estimator and why we are so confident that there will not be a better one?
  2. Many different tutorials say that both $\hat{\beta}_1$ and $\hat{\beta}_2$ are unbiased estimator, but $\hat{\beta}_2$ has smaller variance, so it's better. However, I couldn't figure out how $V(\hat{\beta}_2)$ is smaller than $V(\hat{\beta}_1)$ from above equations. This relation does not look as straightforward as I expected, how to prove this relation?
  3. If we conduct the Wald Test after estimating $\beta$, as following: $$ p = \dfrac{\hat{\beta}^2}{Var(\hat{\beta})} $$ $E[p_2]$ should be greater than $E[p_1]$ because $E[\hat{\beta}_1]=E[\hat{\beta}_2]$ and $V(\hat{\beta}_1)>V(\hat{\beta}_2)$. I tried to show this without usage of $E[y]$, i.e. to show the following smaller than 1: $$ \dfrac{((x^Tx)^{-1}x^Ty)^2}{V(\hat{\beta}_1)} \cdot \dfrac{V(\hat{\beta}_2)}{((x^T\Omega^{-1}x)^{-1}x^T\Omega^{-1}y)^2} $$ and hoping to get rid of $y$, but this seems to lead me to nowhere. I end up with $$ \dfrac{x^Tyy^Tx}{x^T\Omega^{-1}yy^T\Omega^{-1}x}\cdot \dfrac{x^T\Omega^{-1}x}{x^T\Omega x} $$ If $yy^T$ is propotional to $\Omega$, the ratio seems to be one. What's wrong about this? Can anyone please help me?
  • 1
    $\begingroup$ These things are Well expained in 'Basic econometrics' from Gujaratti. $\endgroup$
    – user83346
    Jul 30, 2017 at 15:21

1 Answer 1

  1. We define the BLUE as the one with smallest variance among all linear unbiased estimators under the assumption $\epsilon \sim N(0, \sigma^2I)$. There will not be a better one among this class of estimators by the Gauss-Markov Theorem.

  2. This one is tricky and involves a bit of matrix algebra and manipulation. Note that,

$$\begin{align} \frac{V(\hat{\beta}_1) - V(\hat{\beta}_2)}{\sigma^2} &= (x^Tx)^{-1}x^T\Omega x(x^Tx)^{-1} - (x^T\Omega^{-1} x)^{-1}\\ &=(x^Tx)^{-1}x^T\Omega x(x^Tx)^{-1} - (x^T\Omega^{-1} x)^{-1}x^T\Omega^{-1}\Omega\Omega^{-1}x(x^T\Omega^{-1}x)^{-1}\\ &= \underbrace{[(x^Tx)^{-1}x^T - (x^T\Omega^{-1}x)^{-1}x^T\Omega^{-1}]}_{u^T}\Omega\underbrace{[x(x^Tx)^{-1} - \Omega^{-1}x(x^T\Omega^{-1}x)^{-1}]}_{u} \\ &= u^T\Omega u \\ &> 0\end{align}$$ where the last inequality follows from the positive definite assumption of $\Omega$.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy