In a regression problem, with $y=X\theta+\epsilon$ and $X$ is an $n$ by $p$ matrix the ‘weighted least squares estimate is the minimizer $\theta^{*}$ of $f(\theta)=\sum_{i=1}^{n}\omega_{i}(y_i-x_i^{'}\theta)^2$ for suitable positive ‘weights’ $\omega_i$, where $x_i^{'}$ are the rows of $X$. It can be shown that if $X$ has full rank, then the minimizer $\theta^{*}=(X'WX)^{-1}X'Wy$ where $W=\textrm{diag}(\omega_1,\omega_2,...,\omega_n)$.

Now suppose that the observations are independent, so that the covariance matrix $\Sigma$ of y is diagonal, but that the diagonal elements: $\sigma_1^{2},\sigma_2^{2},...,\sigma_n^{2}$ are not all equal. We say the observations are ‘heteroscedastic’.

Question 1: Show that the covariance matrix of $\theta'$ is given by: $$\operatorname{cov}(\theta^{*})=(X'WX)^{-1}X^{'}W\Sigma WX(X'WX)^{-1}$$ and that the variance of a linear combination $a\theta^{*}$ is: $$\operatorname{Var}[a'\theta^{*}]=a'(X'WX)^{-1}X^{'}W\Sigma WX(X'WX)^{-1}a$$

Question 2:

Show that, for any such linear combination, $\operatorname{Var}[a'\theta^{*}]$ is minimized by the choice of the weights: $\omega_i=1/\sigma_i^2$(i.e, when $W=\Sigma^{-1}$).

I solved question 1, but I have no idea about question 2. I appreciate any help. Thanks!


3 Answers 3


It's early and i'm on the bus...Maybe i'm sleeping, but i tried to solve it:

For $W= S:= \Sigma^{-1}, $

$$a^\top (X^\top SX)^{-1}X^\top S S^{-1} S X(X^\top SX)^{-1} a $$

Where $S^{-1} S = \textrm{diag}(1)$

$$a^\top (X^\top SX)^{-1} X^\top SX(X^\top SX)^{-1}a = a (X^\top SX)^{-1}a .$$

where $(X^\top SX)^{-1} X^\top SX= \textrm{diag}(1)~~~a (X^\top SX)^{-1}a,$ where $(X^\top SX)^{-1}$ is homoscedastic covariance matrix of Theta vector and it is minimal cov matrix for the gauss-markov theorem.


$\require{cancel}$ $$\operatorname{Cov}(\theta^\ast)=E[{\theta^\ast}{\theta^\ast}']-E[{\theta^\ast}]E[{\theta^\ast}]'$$

$$E[{\theta^\ast}]E[{\theta^\ast}]'=\theta \theta'$$

$$y = X\theta+u$$

$$E[{\theta^\ast}{\theta^\ast}']'=E[((X'WX)^{−1}X'Wy)((X'WX)^{−1}X'Wy)']=\\ E[(X'WX)^{−1}X'Wyy'WX(X'WX)^{−1}]=E[(X'WX)^{−1}X'W(X\theta+u)(\theta'X'+u')WX(X'WX)^{−1}]\\ =E[(X'WX)^{−1}X'W(X\theta\theta'X+u\theta'X+X\theta u'+uu')WX(X'WX)^{−1}]= E[\cancel{(X'WX)^{−1}(X'WX)}\theta\theta'\cancel{(XWX)(X'WX)^{−1}}]+\\ \underbrace{E[(X'WX)^{−1}X'W(u\theta'X)WX(X'WX)^{−1}]}_{=0}+\\ \underbrace{E[(X'WX)^{−1}X'W(X\theta u')WX(X'WX)^{−1}]}_{=0}+\\ E[(X'WX)^{−1}X'W(uu')WX(X'WX)^{−1}]\\ =\theta \theta' +(X'WX)^{−1}X'W\overbrace{\Sigma}^{E[uu']} WX(X'WX)^{−1} $$


$$\operatorname{Cov}(\theta^\ast)=\underbrace{\cancel{\theta \theta'} +(X'WX)^{−1}X'W\Sigma WX(X'WX)^{−1}}_{E[{\theta^\ast}{\theta^\ast}']}- \underbrace{\cancel{\theta \theta'}}_{E[{\theta^\ast}]E[{\theta^\ast}]'}\\ =(X'WX)^{−1}X'W\Sigma WX(X'WX)^{−1}$$


$$\operatorname{Var}(a'\theta^\ast)=E[a'\theta^\ast{\theta^\ast}'a]-E[a'\theta^\ast]E[{\theta^\ast}'a]\\ =a'E[\theta^\ast{\theta^\ast}]a - a'E[\theta^\ast]E[\theta^\ast]'a\\ = a'\left(E[\theta^\ast{\theta^\ast}] - E[\theta^\ast]E[\theta^\ast]'\right)a\\ =a'\operatorname{Cov}(\theta^\ast)a$$

If you can use the results of the unweighted version, the result can be directly obtained by recognizing a linear transformation of both $X$ and $y$ that generalizes the weighted linear regression estimator.


There are two methods of showing $\operatorname{Var}(a'\theta^*) \geq \operatorname{Var}(a'\hat{\theta})$, where $\hat{\theta} = (X'\Sigma^{-1}X)^{-1}X'\Sigma^{-1}y$ is the weighted-least squares estimate of $\theta$ with the "theoretical" weights $\Sigma$, and $\theta^* = (X'WX)^{-1}X'Wy$ is an arbitrary weighted-least squares estimate. The first method links the problem to an OLS problem and then applies the Gauss-Markov theorem (as @Danilo attempted but he did not clearly finish the argument). The second method is a brutal-force evaluation of the difference $\operatorname{Var}(a'\theta^*) - \operatorname{Var}(a'\hat{\theta})$.

Method 1

Rewrite the linear model $y = X\theta + \epsilon$ as $y_0 = X_0\theta + u$, where $y_0 = \Sigma^{-1/2}y$, $X_0 = \Sigma^{-1/2}X$, $u = \Sigma^{-1/2}\epsilon$. The latter representation then corresponds to an OLS problem as the error $u$ is homoscedastic in view of $\operatorname{Var}(u) = \Sigma^{-1/2}\Sigma\Sigma^{-1/2} = I_{(n)}$. The Gauss-Markov theorem then applies: since $a'\theta^* = a'(X'WX)^{-1}X'Wy = a'(X'WX)^{-1}X'W\Sigma^{1/2}y_0$ is an unbiased linear estimate of $a'\theta$ (i.e., $E[a'\theta^*] = a'\theta$), it follows that \begin{align} \operatorname{Var}(a'\theta^*) \geq \operatorname{Var}(a'(X_0'X_0)^{-1}X_0'y_0) = \operatorname{Var}(a'\hat{\theta}). \end{align} This completes the proof.

Method 2

Since $\operatorname{Var}(a'\theta^*) - \operatorname{Var}(a'\hat{\theta}) = a'((X'WX)^{-1}X'W\Sigma WX(X'WX)^{-1} - (X'\Sigma^{-1}X)^{-1})a$, if we can show that the matrix $(X'WX)^{-1}X'W\Sigma WX(X'WX)^{-1} - (X'\Sigma^{-1}X)^{-1} \geq 0$ (i.e., the difference is a positive semi-definite matirx), the result then follows. To this end, note that \begin{align} & (X'WX)^{-1}X'W\Sigma WX(X'WX)^{-1} - (X'\Sigma^{-1}X)^{-1} \\ =& (X'WX)^{-1}[X'W\Sigma WX - (X'WX)(X'\Sigma^{-1}X)^{-1}(X'WX)](X'WX)^{-1} \\ =& (X'WX)^{-1}X'W[\Sigma - X(X'\Sigma^{-1}X)^{-1}X']WX(X'WX)^{-1} \\ =& (X'WX)^{-1}X'W\Sigma^{1/2}[I_{(n)} - \Sigma^{-1/2}X(X'\Sigma^{-1}X)^{-1}(\Sigma^{-1/2}X)']\Sigma^{1/2}WX(X'WX)^{-1}, \end{align} hence it suffices to prove $I_{(n)} - \Sigma^{-1/2}X(X'\Sigma^{-1}X)^{-1}(\Sigma^{-1/2}X)' \geq 0$, which follows from the matrix ("hat matrix") $H := \Sigma^{-1/2}X(X'\Sigma^{-1}X)^{-1}(\Sigma^{-1/2}X)'$ is symmetric and idempotent. This completes the proof.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.