6
$\begingroup$

I am implementing a variance-covariance matrix myself and I'm having some trouble understanding what is meant by the expectation this equation:

$ Cov(X, X) = E[(X - E(x))(X - E(x))']$

From what I understand, the expectation of a matrix of variables is the expectation of the columns of the matrix:

$ X = \left( \begin{array}{ccc} 1 & 4 & 7 \\ 2 & 5 & 8 \\ 3 & 6 & 9 \\ 4 & 3 & 2 \end{array} \right) $

$E(X) = \left(\begin{array}{ccc} 2.5 & 4.5 & 6.5 \end{array}\right)' $

I would expect the variance-covariance matrix to be a $3x3$ matrix but using this definition of expectation $(X - E(x))$ is a $4x3$ matrix, $(X - E(x))'$ is a $3x4$ matrix, $(X - E(x))(X - E(x))'$ is therefore a $3x3$ matrix but the the expectation of this is going to be a $3x1$ column vector using the above definition.

How is this outer expectation performed so the result is a $3x3$ matrix?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
6
$\begingroup$

Expectation of a matrix of variables is not the expectation of the columns of the matrix. What may confuse you is that you treat each column as a variable and calculate it's expectation estimate like an average of it's column. In this sense you are right. However covariance matrix is about covariation between this variables. So for example cell (1,3) is covariation between the third and the first variables and so on. Since you have 3 variables it can be 9 covariations (including covariation of variable with itself that is variation) that is how you get $3x3$ matrix.

So covariance matrix is:

$$\begin{bmatrix}Cov(X_{1},X_{1})& Cov(X_{1},X_{2}) &Cov(X_{1},X_{3})\\Cov(X_{2},X_{1})& Cov(X_{2},X_{2}) &Cov(X_{2},X_{3})\\Cov(X_{3},X_{1})& Cov(X_{3},X_{2}) &Cov(X_{3},X_{3})\end{bmatrix}.$$

Finally one more thing. Expectation of random matrix (i.e. matrix which elements are random variables) is a matrix of expectations of it's cells. That is if:

$$X^{*}=\begin{bmatrix}X& Y\\Z& D\\\end{bmatrix},$$

where $X$, $Y$, $Z$ and $D$ are random varibles, then

$$E(X^{*})=\begin{bmatrix}E(X)& E(Y)\\E(Z)& E(D)\\\end{bmatrix}$$

Improve this answer
$\endgroup$
6
  • $\begingroup$ Thanks for your reply - since the expectation of a matrix is the expectation of it's cells, and the cells are all constant wouldn't it become the same matrix then? I apologize if I'm not following. $\endgroup$
    – user124589
    Commented Dec 28, 2016 at 17:43
  • $\begingroup$ The cells are not the constants but also random variables. Suppose independend random variables $x\sim N(1,0)$, $y\sim Bi(10,0.3), z\sim U(0,1)$ and $t=\sim N(15,10),$ then random matrix $v=\begin{bmatrix}x& y\\z & t\end{bmatrix}$ will have expectation: $E(v)=\begin{bmatrix}1& 3\\\frac{1}{2} & 15\end{bmatrix}$ $\endgroup$
    – Bogdan
    Commented Dec 28, 2016 at 17:51
  • $\begingroup$ Also covariation is not random variable as it really constant so it's expectation equals to itself. $\endgroup$
    – Bogdan
    Commented Dec 28, 2016 at 17:53
  • $\begingroup$ thank you for the clarification. Is that why in this post: stats.seandolinar.com/making-a-covariance-matrix-in-r the author creates a "matrix of means" out of the column-wise expectation in order to properly subtract them? $\endgroup$
    – user124589
    Commented Dec 28, 2016 at 18:03
  • $\begingroup$ Exatcly, he just gets E(x) term for covariation matrix formula you have mentioned above. $\endgroup$
    – Bogdan
    Commented Dec 28, 2016 at 18:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.