# Covariance matrix for p dimensional vector

I am working on making a conjecture about necessary and sufficient conditions for a singular covariance matrix of a p-dimensional random vector.

To get to this conjecture I have to find the conditions that the covariance matrix of a 2-dimensional random vector, $$X=(X_1, X_2)^T$$, is singular. Knowing that a matrix is singular if its determinant is 0, I said that the covariance matrix is singular iff $$\sigma_{X_1}^2 \sigma_{X_2}^2 = \sigma_{X_1X_2}^2$$. However, I am not sure how to generalize this to p-dimension.

• Your formula doesn't look like a determinant. Please tell us what you mean by the $\sigma^2_{X_i}.$ What exactly do you need to generalize, given that determinants are well-defined, understood, explained in many texts, and you have already articulated a correct criterion for singularity in any number of dimensions?
– whuber
Commented Jul 29, 2020 at 20:55
• @whuber Thank you for pointing that out for me. I rewrote it. Commented Jul 29, 2020 at 20:56
• Do you mean that $\sigma_{X_1X_2}$ is the covariance? If so, then your equality is equivalent to the determinant being zero.
– whuber
Commented Jul 29, 2020 at 20:57
• @whuber Finding the determinant of a p-dimensional matrix is very tedious. I was hoping there is a simpler way to get the conditions. Commented Jul 29, 2020 at 20:57
• @whuber Yes, $\sigma_{X_1,X_2}$ is the covariance of $X_1, X_2$. Commented Jul 29, 2020 at 20:59

Let's explore the definitions. This will yield a series of simple, effective characterizations of singular covariance matrices.

Suppose $$\Sigma$$ is a covariance matrix for a $$p$$-dimensional random variable $$X = (X_1,X_2,\ldots, X_p)^\prime:$$ $$\Sigma = \operatorname{Cov}(X,X);\quad \Sigma_{ij} = (\operatorname{Cov}(X_i,X_j)).$$ $$\Sigma$$ is singular when there exists a nonzero vector $$a = (a_1,a_2,\ldots,a_p)^\prime$$ for which the variance of $$a^\prime X$$ is zero. This means

There exists a vector $$a$$ for which $$0 = \operatorname{Var}(a^\prime X) = a^\prime\Sigma a.$$

This can be simplified by considering an arbitrary nonzero vector $$\varepsilon$$ and computing the variance of $$(a + t\varepsilon)^\prime X$$ for scalars $$t:$$

\begin{aligned} \operatorname{Var}((a + t\varepsilon)^\prime X) &= \operatorname{Var}(a^\prime X) + 2t\operatorname{Cov}(a^\prime X, \varepsilon^\prime X) + t^2\operatorname{Var}(\varepsilon^\prime X)\\ &= 0 + 2ta^\prime \Sigma\varepsilon + t^2 \operatorname{Var}(\varepsilon^\prime X). \end{aligned}

Because variances, being expectations of squares, cannot be negative, we find

$$0 \le 2t\varepsilon^\prime \Sigma a + t^2\operatorname{Var}(\varepsilon^\prime X).$$

Writing $$\sigma^2 = \operatorname{Var}(\varepsilon^\prime X)$$ and $$b = \varepsilon^\prime \Sigma a$$ permits us to re-express this as

$$0 \le \sigma^2\left( t + \frac{b}{\sigma^2}\right)^2 - \frac{b^2}{\sigma^2}$$

when $$\sigma^2\ne 0$$ and otherwise as

$$0 \le 2bt,$$

inequalities that must hold for all numbers $$t.$$ Since in the first case plugging in $$t = -b/\sigma^2$$ gives $$0 \le -b^2/\sigma^2$$ and in the second case plugging in $$t = -b$$ gives $$0 \le -2b^2,$$ we conclude $$b=0.$$ This shows that

For all vectors $$\epsilon,$$ $$\epsilon^\prime (\Sigma a) = 0,$$ implying $$\Sigma a = 0.$$ Therefore when $$\Sigma$$ is singular, there exists a nonzero $$a$$ with $$\Sigma a = 0.$$

This linear condition on $$a$$ simplifies the previous quadratic condition, $$a^\prime \Sigma a = 0.$$ But since $$\Sigma a = 0$$ implies $$a^\prime \Sigma a = a^\prime 0 = 0,$$ this condition is equivalent to the previous one.

By definition, an eigenvalue of $$\Sigma$$ is a number $$\lambda$$ for which there exists a nonzero vector $$a$$ and $$\Sigma a = \lambda a.$$ (Notice that any eigenvalues of a covariance matrix must be non-negative, because $$0 \le \operatorname{Var}(a^\prime X) = a^\prime \Sigma a = a^\prime (\lambda a) = \lambda ||a||^2$$ can hold only for $$\lambda \ge 0.$$) In this language the preceding condition states

A singular covariance matrix $$\Sigma$$ has a zero eigenvalue.

We can say more. Recall that an invertible matrix is one for which there exists a matrix $$T$$ with $$T\Sigma = 1_p$$ (the unit diagonal matrix). Assuming there is such a $$T$$ would yield the contradiction

$$0 = a^\prime \Sigma a = (a^\prime T)\Sigma a = a^\prime 1_p a = a^\prime a \ne 0.$$

Therefore

Singular matrices $$\Sigma$$ are not invertible.

A basic theorem about determinants is that a matrix is invertible if and only if its determinant is nonzero. Consequently

When $$\Sigma$$ is a singular covariance matrix, $$\det(\Sigma) = 0.$$

Finally, a trivial consequence of this leads to a useful characterization. Suppose there is a subset $$I\subseteq\{1,2,\ldots, p\}$$ of indices for which the covariance matrix of the variables $$X_i$$ with $$i\in I$$ is singular. This covariance matrix consists of the array of entries in $$\Sigma$$ formed from just the rows and columns in $$I,$$ known as a "principal minor." Going back to the original characterization, this means there is some nonzero vector $$a$$ whose only nonzero components are for indices in $$I$$ for which $$0 = a^\prime\Sigma a.$$ But that means $$\Sigma$$ itself is singular.

A covariance matrix $$\Sigma$$ is singular when at least one of its principal minors is singular.

For example, the diagonal elements of $$\Sigma$$ are (considered as $$1\times 1$$ submatrices) principal minors. Thus, the presence of a zero diagonal element implies singularity.

A matrix $$A$$ is a covariance matrix if and only if it is a symmetric positive semi-definite matrix (see here).

A symmetric matrix is positive definite if and only if all of its leading principal minors are strictly positive (see here).

A symmetric matrix is positive semi-definite if and only if all of its principal minors are nonnegative (see here).

I suppose you know what (leading) principal minors are. However, if $$A$$ is a $$n\times n$$ matrix, then (see here):

• a minor is a square submatrix $$A_{IJ}$$ where $$I$$ and $$J$$ are subsets of $$\{1,2,\dots,n\}$$
• a principal minor is the determinant of $$A_{IJ}$$, $$I=J$$;
• a leading principal minor is the determinant of $$A_{IJ}$$ when $$I=J=\{1\}$$, or $$I=J=\{1,2\}$$, or $$I=J=\{1,2,3\}$$, etc.

You need a symmetric matrix which is positive semi-definite, but not positive definite: at least one $$|A_{1,\dots,k;1;\dots,k}|$$, i.e. at least one leading principal minor, must be null.

If $$A$$ is a $$2\times 2$$ matrix, the easiest solution is $$a_{11}=0$$ (the first leading principal minor is null), i.e. if $$X_1$$ is a degenerate random variable, then: $$\begin{vmatrix} 0 & 0 \\ 0 & a \end{vmatrix}=0$$ Another example is: $$Z\sim N(0,1)$$, $$\mathbf{X}=(Z,-Z)$$, because (see here): $$\mathbf{\Sigma}=\begin{bmatrix} 1 & -1 \\ -1, & 1 \end{bmatrix},\quad |\mathbf{\Sigma}|=0$$ i.e. the first leading principal minor is strictly positive, but the second one is null.

In general, degenerate multivariate distributions have singular covariance matrices. See Does $(X,X)'$ follow a bivariate normal distribution?.

• Positive-definiteness is not equivalent to non-singular.
– whuber
Commented Jul 30, 2020 at 13:40
• Right, because a non positive definite matrix may be non-singular: a negative definite matrix, which doesn't make sense as a variance matrix, is non-singular. But a symmetric positive semi-definite matrix which is not positive definite does make sense as a variance matrix and is singular. Isn't it? Commented Jul 30, 2020 at 16:16
• Yes, that's true: but you don't need to apply all of Sylvester's criteria to determine whether the matrix is singular. There are simpler and more efficient solutions. The most efficient I can think of is to triangulize the matrix using Gaussian reduction: as soon as a zero is found on the diagonal you can stop and declare the matrix singular, but if you complete the process with no zeros on the diagonal, the matrix is invertible.
– whuber
Commented Jul 30, 2020 at 17:17
• But as soon as a zero is found on the diagonal I can't stop and declare the matrix positive semi-definite. Singular is not equivalent to positive semi-definiteness, non-singular is not equivalent to positive definiteness :) Commented Jul 30, 2020 at 17:40
• The question concerns determining whether a covariance matrix is singular. Thus, no testing for positive semi-definiteness is needed.
– whuber
Commented Jan 7, 2022 at 18:17