7
$\begingroup$

In estimating vector $\hat{\beta}$ of linear regression, why does variance-covariance matrix of $\hat{\beta}$:

$$V(\hat{\beta})=E[(\hat{\beta} - E(\hat{\beta}))(\hat{\beta} - E(\hat{\beta}))^T]$$

have as the second term inside $E()$ the transpose?

Since I've found covariance to be expressed like (with no transpose):

enter image description here

$\endgroup$

3 Answers 3

8
$\begingroup$

You quote the definition of covariance of two variables, while the form with transpose is the definition of covariance matrix. In first case we are talking about two random variables $X$ and $Y$,

$$ \operatorname{cov}(X,Y) = \operatorname{E}{\big[(X - \operatorname{E}[X])(Y - \operatorname{E}[Y])\big]} $$

while in the second case $\mathbf{X}$ is a vector of random variables $X_1,\dots,X_n$

$$ \mathbf{X} = \begin{bmatrix}X_1 \\ \vdots \\ X_n \end{bmatrix} $$

so we are talking about covariance between multiple variables in form of covariance matrix

$$ \Sigma=\mathrm{E} \left[ \left( \mathbf{X} - \mathrm{E}[\mathbf{X}] \right) \left( \mathbf{X} - \mathrm{E}[\mathbf{X}] \right)^{\rm T} \right] $$

and the transpose appears in here because you are multiplying two vectors.

$\endgroup$
5
$\begingroup$

A covariance matrix is matrix-valued. Covariance of two random variables is an element in that matrix, i.e. a scalar. Hence the covariance formula that you list yields a scalar, while computing $xx^T$ for vector-valued $x$ yields a matrix.

$\endgroup$
5
  • $\begingroup$ So is the reason of transposing that one can do the multiplication inside $E()$? Since the two matrices to be multiplied could be $n \times m$, $n \not = m$. $\endgroup$
    – mavavilj
    Dec 6, 2016 at 17:04
  • $\begingroup$ I don't see how matrix multiplication bears on this question, since you identify $\hat{\beta}$ as a vector, and that's the only portion which bears a transpose. $\endgroup$
    – Sycorax
    Dec 6, 2016 at 17:06
  • $\begingroup$ Isn't $(\hat{\beta}-E(\hat{\beta}))$ a matrix/vector? So in order to calculate $(\hat{\beta}-E(\hat{\beta}))(\hat{\beta}-E(\hat{\beta}))$ one needs to transpose the other. $\endgroup$
    – mavavilj
    Dec 6, 2016 at 17:13
  • $\begingroup$ You're correct; that is how multiplication is defined for vectors. $\endgroup$
    – Sycorax
    Dec 6, 2016 at 18:59
  • $\begingroup$ Also the covariance matrix between X and Y is not the same as the covariance of the parameter estimates. $\endgroup$ Dec 6, 2016 at 23:38
5
$\begingroup$

Small example: Consider a 2x1

$$v = \begin{bmatrix} 7\\ 6 \end{bmatrix}$$

Note that

$$v'v = \begin{bmatrix} 7&6 \end{bmatrix}\begin{bmatrix} 7\\ 6 \end{bmatrix}$$

is a scalar (or 1x1 matrix?)

But

$$vv' = \begin{bmatrix} 7\\ 6 \end{bmatrix} \begin{bmatrix} 7&6 \end{bmatrix}$$

is a matrix

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.