Suppose $y \sim N(\mu, \Sigma)$, where $y \in \mathbb{R}^n$. Let $X \in \mathbb{R}^{n \times p}$ denote a full rank design matrix. By ordinary least squares, the residuals are $$\hat{e} = (I - X(X^TX)^{-1}X^T)y$$ Let $u_1 \in \mathbb{R}^p$ denote a unit vector in $\mathbb{R}^p$. Let $\hat{\beta}_1 = u_1^T (X^TX)^{-1}X^Ty$ denote the OLS estimate for the first covariate in $X$. Suppose I'm interested in the condition distribution of $$\hat{\beta}_1 | \hat{e} = e_0$$

My question is, once I condition on a specific set of the residuals $\hat{e} = e_0$, is there still randomness left in $\hat{\beta}_1$? I'm confused about this because $\hat{\beta}_1$ is a function of $y$, so any variability in $\hat{\beta}_1$ stems from the fact that $y$ is random. Since $\hat{e}$ is a function of $y$, does conditioning on a specific value of $\hat{e}$ imply that there's no more variability left in $\hat{\beta}_1$? In other words, is it possible for different $y$ vectors to give rise to the same residuals, $e_0$?

  • $\begingroup$ Would someone care to elaborate on the down-vote? $\endgroup$
    – Devon
    Mar 28 at 20:57
  • $\begingroup$ Are you still conditioning on $X$ (as in a standard regression problem) or not? $\endgroup$
    – Ben
    Mar 28 at 22:07
  • $\begingroup$ Yes, I'm conditioning on $X$. $\endgroup$
    – Devon
    Mar 29 at 0:46

We know that the vector $\begin{bmatrix} (X^TX)^{-1} X^T \\ I - X (X^TX)^{-1}X^T \end{bmatrix}y$ has a multivariate normal distribution, since it's a linear transformation of a normal distribution. Now, in the case of a multivariate normal, we know that dependence is characterized simply by covariance, so let's investigate the covariance between the first and second block of this multivariable normal distribution.

We will use the result that $\mathrm{cov}(Ay, By) = A \mathrm{cov}(y) B^T$ for conformable matrices $A,B$ and assume that $\mathrm{cov}(y) = \sigma^2 I$. Then the covariance is $$(X^TX)^{-1} X^T \left( \sigma^2 I \right) \left( I - X (X^TX)^{-1} X^T \right) = 0.$$ This means that the blocks are uncorrelated and hence independent. Therefore conditioning on the second half of the block does not change the distribution of the first half the block, i.e. $\hat\beta \stackrel{d}{=} \hat\beta \mid \hat{e}$. So the answer to the question of what is random: all of it.

  • $\begingroup$ This only holds if $cov(y)$ is a diagonal matrix right? $\endgroup$
    – Devon
    Mar 29 at 3:41
  • $\begingroup$ @Devon, it does indeed not hold generally when cov(y) is non-diagonal. A simple counter-example is when you compute the mean $\hat\beta_1 =\hat\mu$ out of a sample of size two. Then $$\hat\mu = 0.5 (y_1+y_2)$$ and $$e_1= y_1-\hat\mu = 0.5 (y_1 - y_2)$$ Their covariance is $$cov(\hat\mu,e_1) = 0.5 var(y_1) -0.5 var(y_2) + cov(y_1,y_2)$$ here you also see that it neither holds when the covariance-matrix is not with identical variances on the diagonal. $\endgroup$ Mar 29 at 8:24
  • $\begingroup$ @SextusEmpiricus The core problem when $\mathrm{cov}(y)=c\Sigma$ is not diagonal is that the OLS estimator doesn't respect the geometry of the model. If you knew $\Sigma$ you could use the MLE of that model, which is sometimes called GLS, and that would have $\hat{e}$ and $\hat\beta$ independent again. $\endgroup$
    – user551504
    Mar 29 at 17:16

To give an explicit example to illustrate that residuals may remain the same even if the dependent variable changes, consider

x <- rep(1:3,2)
y <- c(x[1:3]+1,x[1:3]+3)

y2 <- c(1,3,5,3,5,7)
points(x,y2, col="red")
abline(lm(y2~x), col="red")

enter image description here


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.