# Minimizer weighted linear regression

In a regression problem, with $$y=X\theta+\epsilon$$ and $$X$$ is an $$n$$ by $$p$$ matrix the ‘weighted least squares estimate is the minimizer $$\theta^{*}$$ of $$f(\theta)=\sum_{i=1}^{n}\omega_{i}(y_i-x_i^{'}\theta)^2$$ for suitable positive ‘weights’ $$\omega_i$$, where $$x_i^{'}$$ are the rows of $$X$$. It can be shown that if $$X$$ has full rank, then the minimizer $$\theta^{*}=(X'WX)^{-1}X'Wy$$ where $$W=\textrm{diag}(\omega_1,\omega_2,...,\omega_n)$$.

Now suppose that the observations are independent, so that the covariance matrix $$\Sigma$$ of y is diagonal, but that the diagonal elements: $$\sigma_1^{2},\sigma_2^{2},...,\sigma_n^{2}$$ are not all equal. We say the observations are ‘heteroscedastic’.

Question 1: Show that the covariance matrix of $$\theta'$$ is given by: $$\operatorname{cov}(\theta^{*})=(X'WX)^{-1}X^{'}W\Sigma WX(X'WX)^{-1}$$ and that the variance of a linear combination $$a\theta^{*}$$ is: $$\operatorname{Var}[a'\theta^{*}]=a'(X'WX)^{-1}X^{'}W\Sigma WX(X'WX)^{-1}a$$

Question 2:

Show that, for any such linear combination, $$\operatorname{Var}[a'\theta^{*}]$$ is minimized by the choice of the weights: $$\omega_i=1/\sigma_i^2$$(i.e, when $$W=\Sigma^{-1}$$).

I solved question 1, but I have no idea about question 2. I appreciate any help. Thanks!

It's early and i'm on the bus...Maybe i'm sleeping, but i tried to solve it:

For $$W= S:= \Sigma^{-1},$$

$$a^\top (X^\top SX)^{-1}X^\top S S^{-1} S X(X^\top SX)^{-1} a$$

Where $$S^{-1} S = \textrm{diag}(1)$$

$$a^\top (X^\top SX)^{-1} X^\top SX(X^\top SX)^{-1}a = a (X^\top SX)^{-1}a .$$

where $$(X^\top SX)^{-1} X^\top SX= \textrm{diag}(1)~~~a (X^\top SX)^{-1}a,$$ where $$(X^\top SX)^{-1}$$ is homoscedastic covariance matrix of Theta vector and it is minimal cov matrix for the gauss-markov theorem.

$$\require{cancel}$$ $$\operatorname{Cov}(\theta^\ast)=E[{\theta^\ast}{\theta^\ast}']-E[{\theta^\ast}]E[{\theta^\ast}]'$$

$$E[{\theta^\ast}]E[{\theta^\ast}]'=\theta \theta'$$

$$y = X\theta+u$$

$$E[{\theta^\ast}{\theta^\ast}']'=E[((X'WX)^{−1}X'Wy)((X'WX)^{−1}X'Wy)']=\\ E[(X'WX)^{−1}X'Wyy'WX(X'WX)^{−1}]=E[(X'WX)^{−1}X'W(X\theta+u)(\theta'X'+u')WX(X'WX)^{−1}]\\ =E[(X'WX)^{−1}X'W(X\theta\theta'X+u\theta'X+X\theta u'+uu')WX(X'WX)^{−1}]= E[\cancel{(X'WX)^{−1}(X'WX)}\theta\theta'\cancel{(XWX)(X'WX)^{−1}}]+\\ \underbrace{E[(X'WX)^{−1}X'W(u\theta'X)WX(X'WX)^{−1}]}_{=0}+\\ \underbrace{E[(X'WX)^{−1}X'W(X\theta u')WX(X'WX)^{−1}]}_{=0}+\\ E[(X'WX)^{−1}X'W(uu')WX(X'WX)^{−1}]\\ =\theta \theta' +(X'WX)^{−1}X'W\overbrace{\Sigma}^{E[uu']} WX(X'WX)^{−1}$$

Hence

$$\operatorname{Cov}(\theta^\ast)=\underbrace{\cancel{\theta \theta'} +(X'WX)^{−1}X'W\Sigma WX(X'WX)^{−1}}_{E[{\theta^\ast}{\theta^\ast}']}- \underbrace{\cancel{\theta \theta'}}_{E[{\theta^\ast}]E[{\theta^\ast}]'}\\ =(X'WX)^{−1}X'W\Sigma WX(X'WX)^{−1}$$

Then

$$\operatorname{Var}(a'\theta^\ast)=E[a'\theta^\ast{\theta^\ast}'a]-E[a'\theta^\ast]E[{\theta^\ast}'a]\\ =a'E[\theta^\ast{\theta^\ast}]a - a'E[\theta^\ast]E[\theta^\ast]'a\\ = a'\left(E[\theta^\ast{\theta^\ast}] - E[\theta^\ast]E[\theta^\ast]'\right)a\\ =a'\operatorname{Cov}(\theta^\ast)a$$

If you can use the results of the unweighted version, the result can be directly obtained by recognizing a linear transformation of both $$X$$ and $$y$$ that generalizes the weighted linear regression estimator.

There are two methods of showing $$\operatorname{Var}(a'\theta^*) \geq \operatorname{Var}(a'\hat{\theta})$$, where $$\hat{\theta} = (X'\Sigma^{-1}X)^{-1}X'\Sigma^{-1}y$$ is the weighted-least squares estimate of $$\theta$$ with the "theoretical" weights $$\Sigma$$, and $$\theta^* = (X'WX)^{-1}X'Wy$$ is an arbitrary weighted-least squares estimate. The first method links the problem to an OLS problem and then applies the Gauss-Markov theorem (as @Danilo attempted but he did not clearly finish the argument). The second method is a brutal-force evaluation of the difference $$\operatorname{Var}(a'\theta^*) - \operatorname{Var}(a'\hat{\theta})$$.

#### Method 1

Rewrite the linear model $$y = X\theta + \epsilon$$ as $$y_0 = X_0\theta + u$$, where $$y_0 = \Sigma^{-1/2}y$$, $$X_0 = \Sigma^{-1/2}X$$, $$u = \Sigma^{-1/2}\epsilon$$. The latter representation then corresponds to an OLS problem as the error $$u$$ is homoscedastic in view of $$\operatorname{Var}(u) = \Sigma^{-1/2}\Sigma\Sigma^{-1/2} = I_{(n)}$$. The Gauss-Markov theorem then applies: since $$a'\theta^* = a'(X'WX)^{-1}X'Wy = a'(X'WX)^{-1}X'W\Sigma^{1/2}y_0$$ is an unbiased linear estimate of $$a'\theta$$ (i.e., $$E[a'\theta^*] = a'\theta$$), it follows that \begin{align} \operatorname{Var}(a'\theta^*) \geq \operatorname{Var}(a'(X_0'X_0)^{-1}X_0'y_0) = \operatorname{Var}(a'\hat{\theta}). \end{align} This completes the proof.

#### Method 2

Since $$\operatorname{Var}(a'\theta^*) - \operatorname{Var}(a'\hat{\theta}) = a'((X'WX)^{-1}X'W\Sigma WX(X'WX)^{-1} - (X'\Sigma^{-1}X)^{-1})a$$, if we can show that the matrix $$(X'WX)^{-1}X'W\Sigma WX(X'WX)^{-1} - (X'\Sigma^{-1}X)^{-1} \geq 0$$ (i.e., the difference is a positive semi-definite matirx), the result then follows. To this end, note that \begin{align} & (X'WX)^{-1}X'W\Sigma WX(X'WX)^{-1} - (X'\Sigma^{-1}X)^{-1} \\ =& (X'WX)^{-1}[X'W\Sigma WX - (X'WX)(X'\Sigma^{-1}X)^{-1}(X'WX)](X'WX)^{-1} \\ =& (X'WX)^{-1}X'W[\Sigma - X(X'\Sigma^{-1}X)^{-1}X']WX(X'WX)^{-1} \\ =& (X'WX)^{-1}X'W\Sigma^{1/2}[I_{(n)} - \Sigma^{-1/2}X(X'\Sigma^{-1}X)^{-1}(\Sigma^{-1/2}X)']\Sigma^{1/2}WX(X'WX)^{-1}, \end{align} hence it suffices to prove $$I_{(n)} - \Sigma^{-1/2}X(X'\Sigma^{-1}X)^{-1}(\Sigma^{-1/2}X)' \geq 0$$, which follows from the matrix ("hat matrix") $$H := \Sigma^{-1/2}X(X'\Sigma^{-1}X)^{-1}(\Sigma^{-1/2}X)'$$ is symmetric and idempotent. This completes the proof.