# minimizer weighted linear regression

In a regression problem, with $$y=X\theta+\epsilon$$ and $$X$$ is an $$n$$ by $$p$$ matrix the ‘weighted least squares estimate is the minimizer $$\theta^{*}$$ of $$f(\theta)=\sum_{i=1}^{n}\omega_{i}(y_i-x_i^{'}\theta)^2$$ for suitable positive ‘weights’ $$\omega_i$$, where $$x_i^{'}$$ are the rows of $$X$$. It can be shown that if $$X$$ has full rank, then the minimizer $$\theta^{*}=(X'WX)^{-1}X'Wy$$ where $$W=diag(\omega_1,\omega_2,...,\omega_n)$$.

Now suppose that the observations are independent, so that the covariance matrix $$\Sigma$$ of y is diagonal, but that the diagonal elements: $$\sigma_1^{2},\sigma_2^{2},...,\sigma_n^{2}$$ are not all equal. We say the observations are ‘heteroscedastic’.

Question 1: Show that the covariance matrix of $$\theta'$$ is given by: $$COV(\theta^{*})=(X'WX)^{-1}X^{'}W\Sigma WX(X'WX)^{-1}$$ and that the variance of a linear combination $$a\theta^{*}$$ is: $$Var[a'\theta^{*}]=a'(X'WX)^{-1}X^{'}W\Sigma WX(X'WX)^{-1}a$$

Question 2:

Show that, for any such linear combination, $$Var[a'\theta^{*}]$$ is minimized by the choice of the weights: $$\omega_i=1/\sigma_i^2$$(i.e, when $$W=\Sigma^{-1}$$).

I solved question 1, but I have no idea about question 2. I appreciate any help. Thanks!

## 1 Answer

It's early and i'm on the bus...Maybe i'm sleeping, but i tried to solve it:

For W= S (stands for Sigma^(-1) )

a'(X'SX)^(-1) X'S S^(-1) S X(X'SX)^(-1) a =

Where S^(-1) S = Diag(1)

a'(X'SX)^(-1) X' SX(X'SX)^(-1) a =

a (X'SX)^(-1)a .

where (X'SX)^(-1) X' SX= Diag(1)

a (X'SX)^(-1)a , where (X'SX)^(-1) is homoscedastic covariance matrix of Theta vector and it is minimal cov matrix for the gauss-markov theorem.