# Comparing the estimation of least squares to weight least squares

I was learning the differences between least squares and weighted least squares in terms of estimating $\beta$ for $$y = x\beta + \epsilon$$ where $y$ and $x$ are $n\times 1$ vector and $\epsilon$ is not i.i.d, but $\epsilon \sim N(0, \sigma^2\Omega)$, where $\Omega$ is a $n \times n$ nonsingular, positive definite and symmetric matrix, assumed to be known.

I was learning from these linear regression analysis slides, with a vanilla linear regression (assuming $\epsilon$ i.i.d), we can estimate $\beta$ as $$\hat{\beta}_1 = (x^Tx)^{-1}x^Ty$$ with the expectation and variance as: $$E[\hat{\beta}_1] = (x^Tx)^{-1}x^TE[y] = \beta$$ $$V(\hat{\beta}_1) = (x^Tx)^{-1}x^TV(y)x(x^Tx)^{-1} = \sigma^2(x^Tx)^{-1}x^T\Omega x(x^Tx)^{-1}$$ Then, with weighted least squares, we can estimate $\beta$ as $$\hat{\beta}_2 = (x^T\Omega^{-1}x)^{-1}x^T\Omega^{-1}y$$ with expectation and variance as: $$E[\hat{\beta}_2] = \beta$$ $$V(\hat{\beta}_2) = \sigma^2(x^T\Omega^{-1}x)^{-1}$$ and the author says $\hat{\beta}_2$ is the best linear unbiased estimator of $\beta$

I have these following questions:

1. How to define the best linear unbiased estimator and why we are so confident that there will not be a better one?
2. Many different tutorials say that both $\hat{\beta}_1$ and $\hat{\beta}_2$ are unbiased estimator, but $\hat{\beta}_2$ has smaller variance, so it's better. However, I couldn't figure out how $V(\hat{\beta}_2)$ is smaller than $V(\hat{\beta}_1)$ from above equations. This relation does not look as straightforward as I expected, how to prove this relation?
3. If we conduct the Wald Test after estimating $\beta$, as following: $$p = \dfrac{\hat{\beta}^2}{Var(\hat{\beta})}$$ $E[p_2]$ should be greater than $E[p_1]$ because $E[\hat{\beta}_1]=E[\hat{\beta}_2]$ and $V(\hat{\beta}_1)>V(\hat{\beta}_2)$. I tried to show this without usage of $E[y]$, i.e. to show the following smaller than 1: $$\dfrac{((x^Tx)^{-1}x^Ty)^2}{V(\hat{\beta}_1)} \cdot \dfrac{V(\hat{\beta}_2)}{((x^T\Omega^{-1}x)^{-1}x^T\Omega^{-1}y)^2}$$ and hoping to get rid of $y$, but this seems to lead me to nowhere. I end up with $$\dfrac{x^Tyy^Tx}{x^T\Omega^{-1}yy^T\Omega^{-1}x}\cdot \dfrac{x^T\Omega^{-1}x}{x^T\Omega x}$$ If $yy^T$ is propotional to $\Omega$, the ratio seems to be one. What's wrong about this? Can anyone please help me?
• These things are Well expained in 'Basic econometrics' from Gujaratti.
– user83346
Jul 30, 2017 at 15:21

1. We define the BLUE as the one with smallest variance among all linear unbiased estimators under the assumption $\epsilon \sim N(0, \sigma^2I)$. There will not be a better one among this class of estimators by the Gauss-Markov Theorem.
\begin{align} \frac{V(\hat{\beta}_1) - V(\hat{\beta}_2)}{\sigma^2} &= (x^Tx)^{-1}x^T\Omega x(x^Tx)^{-1} - (x^T\Omega^{-1} x)^{-1}\\ &=(x^Tx)^{-1}x^T\Omega x(x^Tx)^{-1} - (x^T\Omega^{-1} x)^{-1}x^T\Omega^{-1}\Omega\Omega^{-1}x(x^T\Omega^{-1}x)^{-1}\\ &= \underbrace{[(x^Tx)^{-1}x^T - (x^T\Omega^{-1}x)^{-1}x^T\Omega^{-1}]}_{u^T}\Omega\underbrace{[x(x^Tx)^{-1} - \Omega^{-1}x(x^T\Omega^{-1}x)^{-1}]}_{u} \\ &= u^T\Omega u \\ &> 0\end{align} where the last inequality follows from the positive definite assumption of $\Omega$.