In a regression problem, with $y=X\theta+\epsilon$ and $X$ is an $n$ by $p$ matrix the ‘weighted least squares estimate is the minimizer $\theta^{*}$ of $f(\theta)=\sum_{i=1}^{n}\omega_{i}(y_i-x_i^{'}\theta)^2$ for suitable positive ‘weights’ $\omega_i$, where $x_i^{'}$ are the rows of $X$. It can be shown that if $X$ has full rank, then the minimizer $\theta^{*}=(X'WX)^{-1}X'Wy$ where $W=diag(\omega_1,\omega_2,...,\omega_n)$.

Now suppose that the observations are independent, so that the covariance matrix $\Sigma$ of y is diagonal, but that the diagonal elements: $\sigma_1^{2},\sigma_2^{2},...,\sigma_n^{2}$ are not all equal. We say the observations are ‘heteroscedastic’.

Question 1: Show that the covariance matrix of $\theta'$ is given by: $$COV(\theta^{*})=(X'WX)^{-1}X^{'}W\Sigma WX(X'WX)^{-1}$$ and that the variance of a linear combination $a\theta^{*}$ is: $$Var[a'\theta^{*}]=a'(X'WX)^{-1}X^{'}W\Sigma WX(X'WX)^{-1}a$$

Question 2:

Show that, for any such linear combination, $Var[a'\theta^{*}]$ is minimized by the choice of the weights: $\omega_i=1/\sigma_i^2$(i.e, when $W=\Sigma^{-1}$).

I solved question 1, but I have no idea about question 2. I appreciate any help. Thanks!


It's early and i'm on the bus...Maybe i'm sleeping, but i tried to solve it:

For W= S (stands for Sigma^(-1) )

a'(X'SX)^(-1) X'S S^(-1) S X(X'SX)^(-1) a =

Where S^(-1) S = Diag(1)

a'(X'SX)^(-1) X' SX(X'SX)^(-1) a =

a (X'SX)^(-1)a .

where (X'SX)^(-1) X' SX= Diag(1)

a (X'SX)^(-1)a , where (X'SX)^(-1) is homoscedastic covariance matrix of Theta vector and it is minimal cov matrix for the gauss-markov theorem.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.