9
$\begingroup$

I have two sample covariance matrices, computed from $n$ samples, less than $p$ variables: they are singular then.

I know that the sum of two covariance matrices is also a covariance matrix. My question is, if both are singular, is this sum also singular?

And if both are not singular, is this sum also not singular?

$\endgroup$
1
  • 1
    $\begingroup$ Every $n\times n$ covariance matrix or larger can be expressed as the sum of at most $n$ rank-1 (that is, highly singular) covariance matrices. $\endgroup$
    – whuber
    May 20 at 12:31

3 Answers 3

8
$\begingroup$

Sum of singular covariance matrices

No, a sum of singular matrices does not need to be singular.

See Is the sum of two singular matrices also singular?

A counter example in the answers sums two matrices that correspond to matrices for fully positive and fully negative correlation which are each singular but their sum is not.

$$\begin{bmatrix} 1&1\\ 1&1 \end{bmatrix} + \begin{bmatrix} 1&-1\\ -1&1 \end{bmatrix} = \begin{bmatrix} 2&0\\ 0&2 \end{bmatrix} $$

Sum of non-singular covariance matrices

In general it is possible for two non-singular matrices to add up to a singular matrix. E.g. for any non-singular matrix $A$ the matrices $A$ and $B=-A$ are non-singular but the sum $A+B=0$ is singular.

However, as mentioned in the comments this is not true for covariance matrices which are positive definite of the variables are independent. The sum of positive definite matrices, which are non-singular, are positive definite and remain non-singular.

Intuitive approach

If some matrix is a covariance matrix then it has a square root and can be written as $X^tX$. From the definition of the covariance matrix, it is the cross product of vectors after their mean is subtracted.

Then the sum of two covariance matrix can be seen as a single matric where the vectors are concatenated.

The property of singularity can be linked to the independence of the vectors in $X$. As you say, if $X$ has less samples $n$ than variables $p$ then the matrix will be singular. But by adding the samples of two variables together you can get that the variables become independent and the matrix will be non-singular. The other way around can not happen. Once the variables are independent, then you can not get then dependent by adding more samples.

With the example above

$$\overbrace{ \begin{bmatrix} 1\\1 \end{bmatrix}\cdot \begin{bmatrix} 1&1\\ \end{bmatrix} + \begin{bmatrix} 1\\-1 \end{bmatrix}\cdot \begin{bmatrix} 1&-1 \end{bmatrix}}^{\begin{bmatrix} 1&1\\ 1&1 \end{bmatrix} + \begin{bmatrix} 1&-1\\ -1&1 \end{bmatrix} } = \overbrace{ \begin{bmatrix} 1&-1\\ 1&1 \end{bmatrix}\cdot \begin{bmatrix} 1&1\\ -1&1 \end{bmatrix}}^{ \begin{bmatrix} 2&0\\ 0&2 \end{bmatrix} }$$

$\endgroup$
1
  • 6
    $\begingroup$ +1 But your assertion that non-singular (covariance?) matrices can add up to a singular matrix can't be right. For if $A$ and $B$ are both positive definite (and thus non-singular), then $\alpha^T(A+B)\alpha=\alpha^TA\alpha+\alpha^TB\alpha>0$ for all $\alpha\in \mathbb{R}^n\backslash\{0\}$ implying that $A+B$ is also positive definite and non-singular. Also, it is usually the case that $1+1=2$. $\endgroup$ May 20 at 11:46
13
$\begingroup$

The Spectral Theorem informs us that any $n\times n$ covariance matrix $\Sigma$ can be diagonalized. That is, there exists an orthogonal matrix $Q$ for which

$$Q\Sigma Q^\prime = \pmatrix{\sigma^2_1 & 0 & \cdots & 0 \\ 0 & \sigma^2_2 & \cdots & 0 \\ \vdots & \vdots & \ddots & 0\\ 0 & \cdots & 0 & \sigma^2_{n}}.$$

As a mere matter of notation, write $D_{i;n}(\sigma^2)$ for the $n\times n$ matrix that has all zeros except for $\sigma^2$ in its $(i,i)$ position. The sum of these $D_{i;n}(\sigma_i^2)$ over $i=1,2,\ldots, n$ equals this diagonal matrix on the right.

When $X_i$ is a random variable with variance $\sigma_i^2,$ $D_{i;n}(\sigma_i^2)$ is the covariance matrix for the multivariate random variable $\mathbf{X}_i = (0,0,\ldots, 0, X_i,0,\ldots, 0)$ with $X_i$ in the $i^\text{th}$ position. These matrices are obviously singular when $n\gt 1$ (because they have entire rows and columns of zeros); indeed, they manifestly each have rank at most $1$ (the rank is $0$ when $\sigma^2_i=0$). Consequently, every matrix of the form $Q^\prime D_{i;n}(\sigma_i^2) Q$ is singular when $n\gt 1.$

Nevertheless, when the $X_i$ are independent

$$\operatorname{Cov}(Q^\prime(\mathbf{X}_1 + \mathbf{X}_2 + \cdots + \mathbf{X}_n)Q) = Q^\prime D_{1;n}(\sigma^2_1)Q + \cdots + Q^\prime D_{n;n}(\sigma^2_b)Q = \Sigma.$$

You are now well equipped to answer your questions (and any related questions). We have seen how any covariance matrix is the sum of singular covariance matrices, so clearly the sum of singular covariance matrices need not be singular. Nevertheless, because any partial sum of $k\lt n$ matrices in this decomposition has rank at most $k,$ it is singular, showing that the sum of singular covariance matrices can still be singular.

Finally, the sum of nonsingular covariance matrices must be nonsingular due to the positive-definiteness property: when the sum tells you the variance of any vector is zero, then each component of the sum must return zero, too, and positive-definiteness means that this must be the zero vector, proving the sum is positive-definite (and therefore nonsingular).


As a demonstration of practicality, here is an R function to express any covariance matrix $\Sigma$ as a sum of such rank-1 (or rank-0) matrices. It returns the components as a list of $n$ matrices.

decompose <- function(Sigma) {
  zero <- function(d, i) {d[-i] <- 0; d} # Zero out all but element `i` of `d`
  with(svd(Sigma),
    lapply(seq_along(d), function(i) v %*% diag(zero(d, i)) %*% t(v))
  )
}

For instance, let's generate a random covariance matrix, in this case a $4\times 4$ matrix:

set.seed(17)
n <- 4
Sigma <- cov(matrix(rnorm(2*n^2), ncol=n))

After calling decompose, let's re-assemble the components (that is, sum them up) and compare them to the original $\Sigma$:

D <- decompose(Sigma)
Sigma.check <- matrix(0, n, n)
for (d in D) Sigma.check <- Sigma.check + d
round(Sigma.check - Sigma, 8)
     [,1] [,2] [,3] [,4]
[1,]    0    0    0    0
[2,]    0    0    0    0
[3,]    0    0    0    0
[4,]    0    0    0    0

The difference is zero: the decomposition is accurate. Nevertheless, we can verify that each matrix in the decomposition is singular. Since $n$ is small, it suffices to show that all their determinants are zero:

round(sapply(D, det), 16)

[1] 0 0 0 0

And now the acid test: let's generate a lot of the $X_i$ and verify that the covariance matrices of this sample behave as claimed.

library(MASS)
N <- 5e4
X <- lapply(D, function(d) mvrnorm(N, mu=rep(0,n), Sigma=d))

X is an array of $n=4$ samples corresponding to $X_1, \ldots, X_n.$ Each sample should have a nearly singular covariance matrix, so let's look at the singular values of those matrices:

lapply(X, function(x) zapsmall(svd(x)$d))
[[1]]
[1] 230.3927   0.0000   0.0000   0.0000

[[2]]
[1] 193.1368   0.0000   0.0000   0.0000

[[3]]
[1] 156.2236   0.0000   0.0000   0.0000

[[4]]
[1] 103.298   0.000   0.000   0.000

Sure enough, each one of these covariance matrices has only one nonzero singular value: they are all of rank $1$ and all are singular. Finally, let's assemble them into the sum and check that its covariance matrix is close to $\Sigma:$

X.sum <- matrix(0, N, n)
for (x in X) X.sum <- X.sum + x
signif(cov(X.sum) - Sigma, 2)
        [,1]     [,2]    [,3]    [,4]
[1,] -0.0012 -0.00140 -0.0045 -0.0040
[2,] -0.0014 -0.00008  0.0013  0.0035
[3,] -0.0045  0.00130 -0.0042 -0.0045
[4,] -0.0040  0.00350 -0.0045  0.0044

These small differences are due to sampling variation and help confirm everything works as intended.

$\endgroup$
5
$\begingroup$

No, consider the singular covariance matrix $\pmatrix{1 & 0\\ 0 & 0}$ summed with the singular covariance matrix $\pmatrix{0&0\\0&1}$.

Regarding your second question, the sum of two nonsingular covariance matrices is also nonsingular. This is because the set of nonsingular covariance matrices is exactly the set of positive definite matrices (which is closed under positive combination, see here for the sum case).

$\endgroup$
1
  • 2
    $\begingroup$ This is a great intuitive example--considered as the covariances of two MVN distributions, it corresponds to two degenerate axis-aligned 90%-probability "ellipses" (collapsed onto the respective axes). The sum of these two RVs will have identity covariance. $\endgroup$
    – tsbertalan
    May 22 at 15:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.