# Why does variance-covariance matrix of $\hat{\beta}$ have transpose inside?

In estimating vector $\hat{\beta}$ of linear regression, why does variance-covariance matrix of $\hat{\beta}$:

$$V(\hat{\beta})=E[(\hat{\beta} - E(\hat{\beta}))(\hat{\beta} - E(\hat{\beta}))^T]$$

have as the second term inside $E()$ the transpose?

Since I've found covariance to be expressed like (with no transpose):

You quote the definition of covariance of two variables, while the form with transpose is the definition of covariance matrix. In first case we are talking about two random variables $X$ and $Y$,

$$\operatorname{cov}(X,Y) = \operatorname{E}{\big[(X - \operatorname{E}[X])(Y - \operatorname{E}[Y])\big]}$$

while in the second case $\mathbf{X}$ is a vector of random variables $X_1,\dots,X_n$

$$\mathbf{X} = \begin{bmatrix}X_1 \\ \vdots \\ X_n \end{bmatrix}$$

so we are talking about covariance between multiple variables in form of covariance matrix

$$\Sigma=\mathrm{E} \left[ \left( \mathbf{X} - \mathrm{E}[\mathbf{X}] \right) \left( \mathbf{X} - \mathrm{E}[\mathbf{X}] \right)^{\rm T} \right]$$

and the transpose appears in here because you are multiplying two vectors.

A covariance matrix is matrix-valued. Covariance of two random variables is an element in that matrix, i.e. a scalar. Hence the covariance formula that you list yields a scalar, while computing $xx^T$ for vector-valued $x$ yields a matrix.

• So is the reason of transposing that one can do the multiplication inside $E()$? Since the two matrices to be multiplied could be $n \times m$, $n \not = m$. Dec 6, 2016 at 17:04
• I don't see how matrix multiplication bears on this question, since you identify $\hat{\beta}$ as a vector, and that's the only portion which bears a transpose.
– Sycorax
Dec 6, 2016 at 17:06
• Isn't $(\hat{\beta}-E(\hat{\beta}))$ a matrix/vector? So in order to calculate $(\hat{\beta}-E(\hat{\beta}))(\hat{\beta}-E(\hat{\beta}))$ one needs to transpose the other. Dec 6, 2016 at 17:13
• You're correct; that is how multiplication is defined for vectors.
– Sycorax
Dec 6, 2016 at 18:59
• Also the covariance matrix between X and Y is not the same as the covariance of the parameter estimates. Dec 6, 2016 at 23:38

Small example: Consider a 2x1

$$v = \begin{bmatrix} 7\\ 6 \end{bmatrix}$$

Note that

$$v'v = \begin{bmatrix} 7&6 \end{bmatrix}\begin{bmatrix} 7\\ 6 \end{bmatrix}$$

is a scalar (or 1x1 matrix?)

But

$$vv' = \begin{bmatrix} 7\\ 6 \end{bmatrix} \begin{bmatrix} 7&6 \end{bmatrix}$$

is a matrix