I was learning the differences between least squares and weighted least squares in terms of estimating $\beta$ for $$ y = x\beta + \epsilon $$ where $y$ and $x$ are $n\times 1$ vector and $\epsilon$ is not i.i.d, but $\epsilon \sim N(0, \sigma^2\Omega)$, where $\Omega$ is a $n \times n$ nonsingular, positive definite and symmetric matrix, assumed to be known.

I was learning from these linear regression analysis slides, with a vanilla linear regression (assuming $\epsilon$ i.i.d), we can estimate $\beta$ as $$ \hat{\beta}_1 = (x^Tx)^{-1}x^Ty $$ with the expectation and variance as: $$ E[\hat{\beta}_1] = (x^Tx)^{-1}x^TE[y] = \beta $$ $$ V(\hat{\beta}_1) = (x^Tx)^{-1}x^TV(y)x(x^Tx)^{-1} = \sigma^2(x^Tx)^{-1}x^T\Omega x(x^Tx)^{-1} $$ Then, with weighted least squares, we can estimate $\beta$ as $$ \hat{\beta}_2 = (x^T\Omega^{-1}x)^{-1}x^T\Omega^{-1}y $$ with expectation and variance as: $$ E[\hat{\beta}_2] = \beta $$ $$ V(\hat{\beta}_2) = \sigma^2(x^T\Omega^{-1}x)^{-1} $$ and the author says $\hat{\beta}_2$ is the best linear unbiased estimator of $\beta$

I have these following questions:

  1. How to define the best linear unbiased estimator and why we are so confident that there will not be a better one?
  2. Many different tutorials say that both $\hat{\beta}_1$ and $\hat{\beta}_2$ are unbiased estimator, but $\hat{\beta}_2$ has smaller variance, so it's better. However, I couldn't figure out how $V(\hat{\beta}_2)$ is smaller than $V(\hat{\beta}_1)$ from above equations. This relation does not look as straightforward as I expected, how to prove this relation?
  3. If we conduct the Wald Test after estimating $\beta$, as following: $$ p = \dfrac{\hat{\beta}^2}{Var(\hat{\beta})} $$ $E[p_2]$ should be greater than $E[p_1]$ because $E[\hat{\beta}_1]=E[\hat{\beta}_2]$ and $V(\hat{\beta}_1)>V(\hat{\beta}_2)$. I tried to show this without usage of $E[y]$, i.e. to show the following smaller than 1: $$ \dfrac{((x^Tx)^{-1}x^Ty)^2}{V(\hat{\beta}_1)} \cdot \dfrac{V(\hat{\beta}_2)}{((x^T\Omega^{-1}x)^{-1}x^T\Omega^{-1}y)^2} $$ and hoping to get rid of $y$, but this seems to lead me to nowhere. I end up with $$ \dfrac{x^Tyy^Tx}{x^T\Omega^{-1}yy^T\Omega^{-1}x}\cdot \dfrac{x^T\Omega^{-1}x}{x^T\Omega x} $$ If $yy^T$ is propotional to $\Omega$, the ratio seems to be one. What's wrong about this? Can anyone please help me?
  • 1
    $\begingroup$ These things are Well expained in 'Basic econometrics' from Gujaratti. $\endgroup$ – user83346 Jul 30 '17 at 15:21
  1. We define the BLUE as the one with smallest variance among all linear unbiased estimators under the assumption $\epsilon \sim N(0, \sigma^2I)$. There will not be a better one among this class of estimators by the Gauss-Markov Theorem.

  2. This one is tricky and involves a bit of matrix algebra and manipulation. Note that,

$$\begin{align} \frac{V(\hat{\beta}_1) - V(\hat{\beta}_2)}{\sigma^2} &= (x^Tx)^{-1}x^T\Omega x(x^Tx)^{-1} - (x^T\Omega^{-1} x)^{-1}\\ &=(x^Tx)^{-1}x^T\Omega x(x^Tx)^{-1} - (x^T\Omega^{-1} x)^{-1}x^T\Omega^{-1}\Omega\Omega^{-1}x(x^T\Omega^{-1}x)^{-1}\\ &= \underbrace{[(x^Tx)^{-1}x^T - (x^T\Omega^{-1}x)^{-1}x^T\Omega^{-1}]}_{u^T}\Omega\underbrace{[x(x^Tx)^{-1} - \Omega^{-1}x(x^T\Omega^{-1}x)^{-1}]}_{u} \\ &= u^T\Omega u \\ &> 0\end{align}$$ where the last inequality follows from the positive definite assumption of $\Omega$.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.