Covariance matrix for p dimensional vector

Question

I am working on making a conjecture about necessary and sufficient conditions for a singular covariance matrix of a p-dimensional random vector.

To get to this conjecture I have to find the conditions that the covariance matrix of a 2-dimensional random vector, $X=(X_1, X_2)^T$, is singular. Knowing that a matrix is singular if its determinant is 0, I said that the covariance matrix is singular iff $\sigma_{X_1}^2 \sigma_{X_2}^2 = \sigma_{X_1X_2}^2$. However, I am not sure how to generalize this to p-dimension.

Answer 1

Let's explore the definitions. This will yield a series of simple, effective characterizations of singular covariance matrices.

---

Suppose $\Sigma$ is a covariance matrix for a $p$-dimensional random variable $X = (X_1,X_2,\ldots, X_p)^\prime:$ $$\Sigma = \operatorname{Cov}(X,X);\quad \Sigma_{ij} = (\operatorname{Cov}(X_i,X_j)).$$ $\Sigma$ is singular when there exists a nonzero vector $a = (a_1,a_2,\ldots,a_p)^\prime$ for which the variance of $a^\prime X$ is zero. This means

There exists a vector $a$ for which $0 = \operatorname{Var}(a^\prime X) = a^\prime\Sigma a.$

This can be simplified by considering an arbitrary nonzero vector $\varepsilon$ and computing the variance of $(a + t\varepsilon)^\prime X$ for scalars $t:$

$$\begin{aligned} \operatorname{Var}((a + t\varepsilon)^\prime X) &= \operatorname{Var}(a^\prime X) + 2t\operatorname{Cov}(a^\prime X, \varepsilon^\prime X) + t^2\operatorname{Var}(\varepsilon^\prime X)\\ &= 0 + 2ta^\prime \Sigma\varepsilon + t^2 \operatorname{Var}(\varepsilon^\prime X). \end{aligned}$$

Because variances, being expectations of squares, cannot be negative, we find

$$0 \le 2t\varepsilon^\prime \Sigma a + t^2\operatorname{Var}(\varepsilon^\prime X).$$

Writing $\sigma^2 = \operatorname{Var}(\varepsilon^\prime X)$ and $b = \varepsilon^\prime \Sigma a$ permits us to re-express this as

$$0 \le \sigma^2\left( t + \frac{b}{\sigma^2}\right)^2 - \frac{b^2}{\sigma^2}$$

when $\sigma^2\ne 0$ and otherwise as

$$0 \le 2bt,$$

inequalities that must hold for all numbers $t.$ Since in the first case plugging in $t = -b/\sigma^2$ gives $0 \le -b^2/\sigma^2$ and in the second case plugging in $t = -b$ gives $0 \le -2b^2,$ we conclude $b=0.$ This shows that

For all vectors $\epsilon,$ $\epsilon^\prime (\Sigma a) = 0,$ implying $\Sigma a = 0.$ Therefore when $\Sigma$ is singular, there exists a nonzero $a$ with $\Sigma a = 0.$

This linear condition on $a$ simplifies the previous quadratic condition, $a^\prime \Sigma a = 0.$ But since $\Sigma a = 0$ implies $a^\prime \Sigma a = a^\prime 0 = 0,$ this condition is equivalent to the previous one.

By definition, an eigenvalue of $\Sigma$ is a number $\lambda$ for which there exists a nonzero vector $a$ and $\Sigma a = \lambda a.$ (Notice that any eigenvalues of a covariance matrix must be non-negative, because $$0 \le \operatorname{Var}(a^\prime X) = a^\prime \Sigma a = a^\prime (\lambda a) = \lambda ||a||^2$$ can hold only for $\lambda \ge 0.$) In this language the preceding condition states

A singular covariance matrix $\Sigma$ has a zero eigenvalue.

We can say more. Recall that an invertible matrix is one for which there exists a matrix $T$ with $T\Sigma = 1_p$ (the unit diagonal matrix). Assuming there is such a $T$ would yield the contradiction

$$0 = a^\prime \Sigma a = (a^\prime T)\Sigma a = a^\prime 1_p a = a^\prime a \ne 0.$$

Therefore

Singular matrices $\Sigma$ are not invertible.

A basic theorem about determinants is that a matrix is invertible if and only if its determinant is nonzero. Consequently

When $\Sigma$ is a singular covariance matrix, $\det(\Sigma) = 0.$

Finally, a trivial consequence of this leads to a useful characterization. Suppose there is a subset $I\subseteq\{1,2,\ldots, p\}$ of indices for which the covariance matrix of the variables $X_i$ with $i\in I$ is singular. This covariance matrix consists of the array of entries in $\Sigma$ formed from just the rows and columns in $I,$ known as a "principal minor." Going back to the original characterization, this means there is some nonzero vector $a$ whose only nonzero components are for indices in $I$ for which $0 = a^\prime\Sigma a.$ But that means $\Sigma$ itself is singular.

A covariance matrix $\Sigma$ is singular when at least one of its principal minors is singular.

For example, the diagonal elements of $\Sigma$ are (considered as $1\times 1$ submatrices) principal minors. Thus, the presence of a zero diagonal element implies singularity.

Answer 2

A matrix $A$ is a covariance matrix if and only if it is a symmetric positive semi-definite matrix (see here).

A symmetric matrix is positive definite if and only if all of its leading principal minors are strictly positive (see here).

A symmetric matrix is positive semi-definite if and only if all of its principal minors are nonnegative (see here).

I suppose you know what (leading) principal minors are. However, if $A$ is a $n\times n$ matrix, then (see here):

• a minor is a square submatrix $A_{IJ}$ where $I$ and $J$ are subsets of $\{1,2,\dots,n\}$
• a principal minor is the determinant of $A_{IJ}$, $I=J$;
• a leading principal minor is the determinant of $A_{IJ}$ when $I=J=\{1\}$, or $I=J=\{1,2\}$, or $I=J=\{1,2,3\}$, etc.

You need a symmetric matrix which is positive semi-definite, but not positive definite: at least one $|A_{1,\dots,k;1;\dots,k}|$, i.e. at least one leading principal minor, must be null.

If $A$ is a $2\times 2$ matrix, the easiest solution is $a_{11}=0$ (the first leading principal minor is null), i.e. if $X_1$ is a degenerate random variable, then: $$\begin{vmatrix} 0 & 0 \\ 0 & a \end{vmatrix}=0$$ Another example is: $Z\sim N(0,1)$, $\mathbf{X}=(Z,-Z)$, because (see here): $$\mathbf{\Sigma}=\begin{bmatrix} 1 & -1 \\ -1, & 1 \end{bmatrix},\quad |\mathbf{\Sigma}|=0$$ i.e. the first leading principal minor is strictly positive, but the second one is null.

In general, degenerate multivariate distributions have singular covariance matrices. See Does $(X,X)'$ follow a bivariate normal distribution?.