4
$\begingroup$

I'm trying to understand the expression $Cov(\hat y,\hat \epsilon)$ in regards to the usual linear regression model/assumptions $y = \beta_0 + \beta_1 x_1 + \beta_2 x_2 + ... \beta_n x_n +\epsilon$. Also, $x_i$ are not considered to be random variables. Rather, they are considered to be fixed measurements with negligible error.

Each $\hat y_i$ is the sum of the $x_i$s weighted by the $\hat \beta_i$s (with $x_0=1)$. And each $\hat \epsilon_i$ is the difference ($y_i - \hat y_i$).

Are $\hat y,\hat \epsilon$, and for that matter, $\hat \beta,$ random vectors?

If u,v are random vectors, then $Cov(u,v)$ is the matrix of elements $Cov(u_i,v_j)$

If $u,v$ are not random vectors, then $Cov(u,v)$ is the scalar $\Sigma u_i v_i$.

Improve this question
$\endgroup$
5
  • 3
    $\begingroup$ One way to tell a random vector from a non-random one is to contemplate what changes would occur if the data-collection process were replicated. Any vector that could change must do so due to the random changes modeled by $\epsilon$, because all other values--the $x_i$ and $\beta_j$--are modeled as constants. BTW, your last line (which asserts the covariance of two vectors is their dot product) isn't remotely close to any standard definition of a "covariance," whether of data or random variables. $\endgroup$
    – whuber ♦
    Oct 26, 2015 at 20:01
  • $\begingroup$ @whuber - I thought that the dot-product of two vectors equated to their covariance? If the dot-product equaled zero then they are orthogonal, and hence independent, and hence their covariance=0? $\endgroup$
    – cwackers
    Oct 26, 2015 at 22:24
  • 2
    $\begingroup$ Not quite. When the dot product of the recentered vectors is zero, they are considered "orthogonal" in the sense of having zero covariance--but that does not mean that the dot product equals the covariance! If you look up any formula for covariance, you will see the difference. $\endgroup$
    – whuber ♦
    Oct 26, 2015 at 22:26
  • $\begingroup$ @whuber Is there a contradiction between your statement "$...\beta_j$ are modeled as constants" and the statement in the answer below "... the betas are random vectors"? Just trying to learn from you. I liked your succinct explanation very much, but this is a loose end. $\endgroup$
    – Antoni Parellada
    Oct 27, 2015 at 12:11
  • $\begingroup$ @l'ombradel'atzavara That answer uses different terminology than the question: its "betas" are the estimates $\hat\beta_j$. $\endgroup$
    – whuber ♦
    Oct 27, 2015 at 13:07

1 Answer 1

Reset to default
4
$\begingroup$

Your last assertion is not true. If two vectors are not random, then their covariance is zero.

You are right nevertheless that the fitted values, the residuals and the betas are random vectors. The reason for this is that they are all linear combinations of the random $\mathbf{y}$. To see this we are going to need to define the projection matrix and its orthogonal complement. The projection matrix is defined as

$$\mathbf{H}= \mathbf{X} \left(\mathbf{X}^{\prime} \mathbf{X} \right)^{-1} \mathbf{X} ^{\prime}$$

and its orthogonal complement, also referred to as the annihilator, is

$$\mathbf{M}= \mathbf{I}_n- \mathbf{H}$$

Using these matrices, we may represent the fitted values and the residuals as

$$\widehat{\mathbf{y}}= \mathbf{H} \mathbf{y}$$

$$\mathbf{e} = \mathbf{M} \mathbf{y}$$

respectively, while the beta vector is given by

$$\mathbf{b} =\left(\mathbf{X}^{\prime} \mathbf{X} \right)^{-1} \mathbf{X} ^{\prime} \mathbf{y}$$

and all are linear combinations of the random response vector, as claimed. It can also be shown that the covariance between the residuals and the betas and consequently, the fitted values, is zero.

Is that what you were looking for?

Improve this answer
$\endgroup$
3
  • 1
    $\begingroup$ +1 for helping clarify these concepts. I have a question regarding nomenclature: I thought that $H = X(X'X)^{-1}X'$ was the "hat matrix", whereas the "projection matrix" was defined as $P = (X'X)^{-1}X'$ so that the vector $y$ would be projected onto the column space of the $X$ by virtue of $P$, giving us the $\beta$ coefficients. So $APy = Hy = fitted\,values$. As I said, it is pure terminology, and I learned a lot from your nice post, but if you could clarify all this, I would really appreciate it. $\endgroup$
    – Antoni Parellada
    Oct 26, 2015 at 20:41
  • 1
    $\begingroup$ @l'ombradel'atzavara Thanks. The projection matrix is exactly defined like $\mathbf{H}$, the hat matrix, even in books of linear algebra. I am using Strang for instance. You can check that it has all the properties of a projection matrix, namely symmetry and idempotence, while your definition of it does not. $\endgroup$
    – JohnK
    Oct 26, 2015 at 20:44
  • $\begingroup$ You are right. $\hat{\mathbf{y}} = X \hat{\boldsymbol \beta} = X \left(X^\top X \right)^{-1} X^\top \mathbf{y}$. And the hat matrix is: $H = X \left(X^\top X \right)^{-1} X^\top$. $\endgroup$
    – Antoni Parellada
    Oct 26, 2015 at 21:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.