Is the covariance matrix almost always postive definite?

Question

I understand a covariance matrix is always positive semi-definite, but it seems that the covariance matrix would almost always be positive definite (although theoretically is only guaranteed to be positive semi-definite).

For example, $\det [A] \geq 0$ if $A$ is positive semi-definite, and $\det [A] > 0$ if $A$ is positive definite. Since the product of eigenvalues is the determinant, the implication of $\det [A] > 0$ is that no eigenvalue is zero. In other words, if a covariance matrix is positive definite, variability exists on all variables no matter how we transform the data.

Is this reasoning correct? Is this not almost always the case since data is never perfect? If so, are most covariance matrices actually positive definite?

Answer 1

As @whuber said, as soon as we step into a multivariate setting, it is likely we have features that they reflect similar sources of variability. As we aggregate additional features in our analysis it becomes more likely we have collinearities and thus end up with PSD matrices.

I think the flaw in the presenting reasoning stems from your point that "variability exists on all variables no matter how we transform the data." We can have variability in all of our data features but the crux is that this variability is potentially reflecting identical information.

For example see the R code below:

set.seed(123)
x1 = runif(4)
x2 = runif(3)
A  = x1 %*% t(x2) # Obviously a rank-deficient matrix // Rank-1 matrix

While the data matrix A is of rank 1 (i.e. it has a single non-zero eigenvalue), every feature of the matrix A is having some variability.

apply(A, 2, var)
# [1] 0.08313244 0.33252977 0.74819198

The point is that all columns are perfectly correlated with each other so the covariance matrix is singular.

cor(A)
#      [,1] [,2] [,3]
# [1,]    1    1    1
# [2,]    1    1    1
# [3,]    1    1    1