Variance matrix of a sum of random vectors

Let X and Y be random vectors of same dimension.

Let var(X) be the covariance matrix of X; var(Y) the correspondent matrix; cov(X,Y) the matrix where the coordinate (i,j) is cov(x_i, y_j)

I saw the affirmation that var(X+Y) is var(X)+ var(Y) + 2 cov(X,Y), and have been trying to prove it.

For positions (i,i), the fact seems simple enough. Take, however, the position (1,2). Making A=X+Y, that position it is supposed to be

$$\text{cov}(a_1,a_2)$$ $$E [(x_1+y_1-\mu_{x_1}-\mu_{y_1}) (x_2+y_2-\mu_{x_2}-\mu_{y_2})]$$ $$E (x_1-\mu_{x_1} + y_1-\mu_{y_1}) (x_2-\mu_{x_2} + y_2-\mu_{y_2})$$ $$E (x_1-\mu_{x_1})(x_2-\mu_{x_2})+ E(x_1-\mu_{x_1})(y_2-\mu_{y_2})+ E(y_1-\mu_{y_1}) (x_2-\mu_{x_2}) + E(y_1-\mu_{y_1})(y_2-\mu_{y_2})$$ $$\text{cov}(x_1,x_2)+ \text{cov}(x_1,y_2) + \text{cov}(x_2,y_1) + \text{cov}(y_1,y_2)$$

$$\text{cov}(x_1,x_2)+ \text{cov}(y_1,y_2) + \text{cov}(x_1,y_2) + \text{cov}(x_2,y_1)$$

The first two terms come from the covariance matrixes of x and y, but the last two seem to fit 2*cov(x,y) only if that last matrix is symetric (and there does not seem to be any reason for this to be the case...)

The variance covariance matrix is defined by \begin{align} \mathbb E[(X+Y-\mu_X-\mu_Y)&(X+Y-\mu_X-\mu_Y)^\top]\\ &= \mathbb E[(X-\mu_X+Y-\mu_Y)(X-\mu_X+Y-\mu_Y)^\top] \\ &= \mathbb E[(X-\mu_X)(X-\mu_X)^\top]\\ &\quad + \mathbb E[(Y-\mu_Y)(Y-\mu_Y)^\top]\\ &\quad + \mathbb E[(Y-\mu_Y)(X-\mu_X)^\top]\\ &\quad + \mathbb E[(X-\mu_X)(Y-\mu_Y)^\top]\\ &= \mathbb E[(X-\mu_X)(X-\mu_X)^\top]\\ &\quad + \mathbb E[(Y-\mu_Y)(Y-\mu_Y)^\top]\\ &\quad + 2\mathbb E[(Y-\mu_Y)(X-\mu_X)^\top]\qquad\text{[symmetry]}\\ &= \operatorname{Var}(X)+\operatorname{Var}(Y)+2\operatorname{Cov}(X,Y) \end{align}

Edit:

The last step is wrong, because in this question X and Y are random vectors, not random variables.

The covariance for two random variables is "symmetric", because $$\operatorname{Cov}(x,y):=\mathbb E[(x-\mathbb E(x))(y-\mathbb E(y))]=\mathbb E[xy]-\mathbb E[x] \mathbb E[y]=\operatorname{Cov}(y,x)$$ However, the cross covariance matrix for two random vectors is not necessarily symmetric, $$\operatorname{Cov}(X,Y):=\mathbb E[(X-\mathbb E(X))(Y-\mathbb E(Y))^\top]=\mathbb E[XY^\top]+\mathbb E[X]\mathbb E[Y^\top]\\$$ Note that we do have this relationship, $$\operatorname{Cov}(X,Y):=\mathbb E[(X-\mathbb E(X))(Y-\mathbb E(Y))^\top]\\ =\mathbb E[((Y-\mathbb E(Y))(X-\mathbb E(X))^\top)^\top]\\ =(\mathbb E[(Y-\mathbb E(Y))(X-\mathbb E(X))^\top])^\top\\ =\operatorname{Cov}(Y,X)^\top\\$$ Therefore, \begin{align} &\mathbb E[(X-\mu_X)(X-\mu_X)^\top]\\ &\quad + \mathbb E[(Y-\mu_Y)(Y-\mu_Y)^\top]\\ &\quad + \mathbb E[(Y-\mu_Y)(X-\mu_X)^\top]\\ &\quad + \mathbb E[(X-\mu_X)(Y-\mu_Y)^\top]\\ &= \operatorname{Var}(X)+\operatorname{Var}(Y)+\operatorname{Cov}(Y,X)+\operatorname{Cov}(X,Y)\\ &= \operatorname{Var}(X)+\operatorname{Var}(Y)+\operatorname{Cov}(X,Y)^\top+\operatorname{Cov}(X,Y).\\ \end{align}

• why is that matrix symmetrical, though? Oct 19, 2020 at 13:24
• Hint: take one element at a time. Oct 19, 2020 at 13:39
• Sorry, I seem to be really stuck :P In the matrix E[(Y-mu_Y)(X-mu_X)^t] the position (1,2) is E(Y1-mu_y1)(X2-mu_x2) = cov(Y_1,X_2), right? Oct 19, 2020 at 13:45
• While the position (2,1) is E(Y2-mu_y2)(X1-mu_x1) = cov(Y_2,X_1), right? Oct 19, 2020 at 13:46

Your notation causes the confusion. If you use two subscripts for the individual elements, it becomes much simpler to show. Assuming the means are zero ( for ease of notation ):

$$E(x_{(1,1)} + y_{(1,2)})^2 =$$

$$E(x_{(1,1)}^2) + E(x_{(1,1)}y_{(1,2)}) + E(y_{(1,2)}x_{(1,1)}) + E(y_{1,2}^2) =$$

$$Var(x_{(1,1)}) + Var(y_{(1,2)}) + 2 \times cov(y_{(1,2)}, x_{(1,1)})$$

• I must have some fundamental misunderstanding... X and Y are in Rn, right? Why would I use two indices? Oct 19, 2020 at 13:25
• Hi: Maybe my answer was more confusing than helpful. You really don't need two indices. By (1,1), I was referrring to the first x element in the $X$ vector and, by (1,2), I was referring to the second element in the $Y$ vector. So, when I wrote above, it is for two specific elements of each vector and not the whole vector. But the result of it goes in the [first row, second column] and in the [second row, first column] positions of the covariance matrix. So that was why I gave them the two subscripts. I hope it clarifies the confusion. Oct 19, 2020 at 19:31
• Hi: One more thing. Note that, if you do the same thing for all the elements, you will obtain an $n \times n$ covariance matrix so yes, it's consistent with both vectors being in $R^{n}$. I just did it for two specific elements of the $R^n$ vectors. Oct 19, 2020 at 19:36
• Note that Xian did it all at once very nicely so you're probably better off considering his answer. The symmetry ( in the scalar case) is due to the fact that cov(x,y) = E(xy) - E(x)E(y). This formula shows that the order is irrelevant which gives you the symmetry. Oct 19, 2020 at 19:39