I'm currently self studying linear regression. Following is an entrance exam problem of a graduate school. Consider the regression model with usual assumptions of the errors $y=X\beta+\epsilon$. Show that $\hat{β}$ and $e$ are statistically independent. $\hat{β}$ is the least square estimator and $e$ is the residual.) Honestly I don't even heard about independence of two random vectors. (I searched my text book of mathematical statistics and linear regression but I couldn't.) So how should I approach it? Thanks in advance. Source of the problem is graduate entrance exam of Seoul National University of 2014

  • $\begingroup$ Please add the [self-study] tag & read its wiki. $\endgroup$ Feb 26, 2015 at 5:23
  • $\begingroup$ Is $\epsilon$ multivariate normal? What about $\hat{\beta}$? What do you know about independence of components of a multivariate normal random vector and the covariance matrix? Can you compute the covariance between $e$ and $\hat{\beta}$? Can you at least get $\hat{\beta}$ and $e$ into the same equation? $\endgroup$ Feb 26, 2015 at 5:51

1 Answer 1

  1. Two Random vectors $\textbf{x},\textbf{y}$ are independant iff $\forall i,j \in 1...n$, $\textbf{x}_{i}$ and $\textbf{y}_{j}$ are independant.
  2. Cross covariance matrix is defined by $cov(x,y )= E[(\textbf{x} - \mu_{\textbf{x}})(\textbf{y} - \mu_{\textbf{y}})^{T}]$
  3. two multivariate normal random vectors are independent iff their cross covariance matrix is zero matrix. (Since each component of Cross covariance matrix is covariance of .. and two normal random variable's covariance=0 iff they are independant)

  4. under the assumption that distribution of $\epsilon$ is $N(0,\sigma I)$ and $\widehat{\beta}$ is the ordinary least square esitimator $Hy$, if one compute covariance matrix of $\widehat{\beta}$ and $\textbf{e}$ , it is zero. Therefore they are statistically independant.

  5. Am I right? thank you Brian Borchers, your commment was very helpful.
  • $\begingroup$ Not sure if zero covariance implies independence in this setting. Zero covariance certainly does not imply independence in general; is there something special in this setting? $\endgroup$ Sep 9, 2021 at 18:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.