1
$\begingroup$

$$cov\begin{bmatrix}\bar{y}\\\hat{\beta}_c\end{bmatrix}=\sigma^2\begin{bmatrix}\frac{1}{n} & 0^T\\0 & (X_c^TX_c)^{-1}\end{bmatrix}$$ with $\hat{\beta}_c=(X_c^TX)^{-1}X^T_cy$.

I am supposed to show two estimators are independent. I know I can show that the covariance part of the matrix is 0 but I didn't learn matrix theory back in collage. I am wondering if someone could show me how the calculation is conducted so I can have a rough idea of what should be done?

$\endgroup$
0
1
$\begingroup$

Let me start from the beginning. You have a model $$y=X\beta+\epsilon=\beta_0+\beta_1x_1+\dots+\beta_px_p+\epsilon$$ where $\epsilon\sim\mathcal{N}(0,\sigma^2I)$, $y\sim\mathcal{N}(X\beta,\sigma^2I)$, and $\hat\beta=(X^TX)^{-1}X^Ty$. Il you center your independent variables, you get: $$y=\beta_0+\beta_1(x_1-\bar{x}_1)+\dots+\beta_p(x_p-\bar{x}_p)+\epsilon=\tilde{X}\beta+\epsilon$$ where $\tilde{X}=(1,X_c)$ and $X_c$ has typical element $x_{ij}-\bar{x}_j$. The estimated coefficients are: $$\hat\beta=(\hat\beta_0,\beta_c),\qquad\hat\beta_0=\bar{y},\qquad \hat\beta_c=(X_c^TX_c)^{-1}X_c^Ty$$ In general, when $y$ is a random vector and $C$ is a matrix, $\text{cov}(Cy)=C\text{cov}(y)C^T$. If $\hat\beta=(X^TX)^{-1}X^Ty$ then, since $X^TX$ is symmetric: \begin{align*} \text{cov}(\hat\beta)&=(X^TX)^{-1}X^T\text{cov}(y)[(X^TX)^{-1}X^T]^T \\ &=(X^TX)^{-1}X^T\sigma^2X(X^TX)^{-1}\\ &=\sigma^2(X^TX)^{-1}(X^TX)(X^TX)^{-1}=\sigma^2(X^TX)^{-1} \end{align*} Let's now consider the simpler model $y=\beta_0+\beta_1x$, where $x=(x_1,x_2,x_3)=(1,2,3)$. The $X^TX$ matrix is: \begin{align*} X^TX&=\begin{bmatrix} 1 & 1 & 1 \\ 1 & 2 & 3 \end{bmatrix}\begin{bmatrix} 1 & 1 \\ 1 & 2 \\ 1 & 3 \end{bmatrix} =\begin{bmatrix} \sum_j 1 & \sum_j1x_{j}\\ \sum_jx_{2}^T1 & \sum_jx_{j}^Tx_{j}\end{bmatrix}\\ &=\begin{bmatrix} n & \sum_j x_j \\ \sum_j x_j & \sum_j x_j^2 \end{bmatrix}=\begin{bmatrix}3 & 6 \\ 6 & 14 \end{bmatrix} \end{align*} Its inverse is \begin{align*} (X^TX)^{-1}&=\frac{1}{n\sum_jx_j^2-\left(\sum_jx_j\right)^2} \begin{bmatrix} \sum_jx_j^2 & -\sum_jx_j \\ -\sum_jx_j & n \end{bmatrix}\\ &=\begin{bmatrix}\frac{1}{n}+\frac{\bar{x}^2}{\sum_j(x_j-\bar{x})^2} & -\frac{\sum_jx_j}{n\sum_jx_j^2-\left(\sum_jx_j\right)^2} \\ -\frac{\sum_jx_j}{n\sum_jx_j^2-\left(\sum_jx_j\right)^2} & \frac{1}{\sum_j(x_j-\bar{x})^2} \end{bmatrix} =\frac16\begin{bmatrix}14 & -6 \\ -6 & 3\end{bmatrix}=\begin{bmatrix}2.\bar{3} & -1 \\ -1 & 0.5 \end{bmatrix} \end{align*} If you replace $X$ with $\tilde{X}=(1,X_c)$, then $\sum_jx_j=0$ and \begin{align*} \tilde{X}^T\tilde{X}&=\begin{bmatrix} 1 & 1 & 1 \\ -1 & 0 & 1 \end{bmatrix}\begin{bmatrix} 1 & -1 \\ 1 & 0 \\ 1 & 1 \end{bmatrix}=\begin{bmatrix} 3 & 0 \\ 0 & 2\end{bmatrix}\\ (\tilde{X}^T\tilde{X})^{-1}&=\begin{bmatrix} \frac13 & 0 \\ 0 & \frac12\end{bmatrix} \end{align*} In general (see Seber & Lee, Linear Regression Analysis, John Wiley & Sons, 2003, p. 120), $$(X^TX)^{-1}=\begin{bmatrix}\frac1n+\bar{x}^TV^{-1}\bar{x} & -\bar{x}^TV^{-1} \\ -V^{-1}\bar{x} & V^{-1}\end{bmatrix}$$ where $\bar{x}$ is a vector of means and $V=X_c^TX_c$. If $X=\tilde{X}$, then $\bar{x}$ is a null vector and $$(\tilde{X}^T\tilde{X})^{-1}=\begin{bmatrix}\frac1n & 0 \\ 0 & (X_c^TX_c)^{-1}\end{bmatrix}$$ Therefore $\hat\beta_0=\bar{y}$ and $\hat\beta_c$ are uncorrelated.

HTH

PS: You can also look at Linear regression $y_i=\beta_0 + \beta_1x_i + \epsilon_i$ covariance between $\bar{y}$ and $\hat{\beta}_1$, where linear algebra is not used.

$\endgroup$
4
  • $\begingroup$ Thank you so much for your explanation. How about this post:stats.stackexchange.com/questions/214539/… ? $\endgroup$
    – JoZ
    Sep 13 '20 at 19:56
  • $\begingroup$ Actually I am trying to use this way of thinking to demonstrate the whole picture of the question, which I have posted here:stats.stackexchange.com/questions/487303/… $\endgroup$
    – JoZ
    Sep 13 '20 at 19:57
  • $\begingroup$ But I have encountered problems which I cannot decern myself... $\endgroup$
    – JoZ
    Sep 13 '20 at 19:57
  • $\begingroup$ I'm going to post an answer. $\endgroup$
    – Sergio
    Sep 13 '20 at 22:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.