10
$\begingroup$

Suppose I have covariance matrices $X$ and $Y$. Which of these options are then also covariance matrices?

  1. $X+Y$
  2. $X^2$
  3. $XY$

I have a bit of trouble understanding what exactly is needed for something to be a covariance matrix. I suppose it is meant that for instance if $X=\operatorname{cov}(X_1,X_2)$, and $Y=\operatorname{cov}(Y_1,Y_2)$ that for 1 to hold true we should have that $\operatorname{cov}(X_1,X_2) + \operatorname{cov}(Y_1,Y_2) = \operatorname{cov}(Z_1, Z_2)$, where $Z_1$ and $Z_2$ are some other random variables. However, I can't see why that would hold true for any of the three options. Any insight would be apprciated.

$\endgroup$
12
$\begingroup$

Background

A covariance matrix $\mathbb{A}$ for a vector of random variables $X=(X_1, X_2, \ldots, X_n)^\prime$ embodies a procedure to compute the variance of any linear combination of those random variables. The rule is that for any vector of coefficients $\lambda = (\lambda_1, \ldots, \lambda_n)$,

$$\operatorname{Var}(\lambda X) = \lambda \mathbb{A} \lambda ^\prime.\tag{1}$$

In other words, the rules of matrix multiplication describe the rules of variances.

Two properties of $\mathbb{A}$ are immediate and obvious:

  1. Because variances are expectations of squared values, they can never be negative. Thus, for all vectors $\lambda$, $$0 \le \operatorname{Var}(\lambda X) = \lambda \mathbb{A} \lambda ^\prime.$$ Covariance matrices must be non-negative-definite.

  2. Variances are just numbers--or, if you read the matrix formulas literally, they are $1\times 1$ matrices. Thus, they do not change when you transpose them. Transposing $(1)$ gives $$\lambda \mathbb{A} \lambda ^\prime = \operatorname{Var}(\lambda X) = \operatorname{Var}(\lambda X) ^\prime = \left(\lambda \mathbb{A} \lambda ^\prime\right)^\prime = \lambda \mathbb{A}^\prime \lambda ^\prime.$$ Since this holds for all $\lambda$, $\mathbb{A}$ must equal its transpose $\mathbb{A}^\prime$: covariance matrices must be symmetric.

The deeper result is that any non-negative-definite symmetric matrix $\mathbb{A}$ is a covariance matrix. This means there actually is some vector-valued random variable $X$ with $\mathbb{A}$ as its covariance. We may demonstrate this by explicitly constructing $X$. One way is to notice that the (multivariate) density function $f(x_1,\ldots, x_n)$ with the property $$\log(f) \propto -\frac{1}{2} (x_1,\ldots,x_n)\mathbb{A}^{-1}(x_1,\ldots,x_n)^\prime$$ has $\mathbb{A}$ for its covariance. (Some delicacy is needed when $\mathbb{A}$ is not invertible--but that's just a technical detail.)

Solutions

Let $\mathbb{X}$ and $\mathbb{Y}$ be covariance matrices. Obviously they are square; and if their sum is to make any sense they must have the same dimensions. We need only check the two properties.

  1. The sum.

    • Symmetry $$(\mathbb{X}+\mathbb{Y})^\prime = \mathbb{X}^\prime + \mathbb{Y}^\prime = (\mathbb{X} + \mathbb{Y})$$ shows the sum is symmetric.
    • Non-negative definiteness. Let $\lambda$ be any vector. Then $$\lambda(\mathbb{X}+\mathbb{Y})\lambda^\prime = \lambda \mathbb{X}\lambda^\prime + \lambda \mathbb{Y}\lambda^\prime \ge 0 + 0 = 0$$ proves the point using basic properties of matrix multiplication.
  2. I leave this as an exercise.

  3. This one is tricky. One method I use to think through challenging matrix problems is to do some calculations with $2\times 2$ matrices. There are some common, familiar covariance matrices of this size, such as $$\pmatrix{a & b \\ b & a}$$ with $a^2 \ge b^2$ and $a \ge 0$. The concern is that $\mathbb{XY}$ might not be definite: that is, could it produce a negative value when computing a variance? If it will, then we had better have some negative coefficients in the matrix. That suggests considering $$\mathbb{X} = \pmatrix{a & -1 \\ -1 & a}$$ for $a \ge 1$. To get something interesting, we might gravitate initially to matrices $\mathbb{Y}$ with different-looking structures. Diagonal matrices come to mind, such as $$\mathbb{Y} = \pmatrix{b & 0 \\ 0 & 1}$$ with $b\ge 0$. (Notice how we may freely pick some of the coefficients, such as $-1$ and $1$, because we can rescale all the entries in any covariance matrix without changing its fundamental properties. This simplifies the search for interesting examples.)

    I leave it to you to compute $\mathbb{XY}$ and test whether it always is a covariance matrix for any allowable values of $a$ and $b$.

$\endgroup$
13
$\begingroup$

A real matrix is a covariance matrix if and only if it is symmetric positive semi-definite.

Hints:

1) If $X$ and $Y$ are symmetric, is $X+Y$ symmetric? If $z^TX z \ge 0$ for any $z$ and $z^TY z \ge 0$ for any $z$, what can you conclude about $z^T(X+Y)z$?

2) If $X$ is symmetric, is $X^2$ symmetric? If the eigenvalues of $X$ are non-negative, what can you conclude about the eigenvalues of $X^2$?

3) If $X$ and $Y$ are symmetric, can you conclude that $XY$ is symmetric, or can you find a counter-example?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.