Is the covariance matrix almost always postive definite?

I understand a covariance matrix is always positive semi-definite, but it seems that the covariance matrix would almost always be positive definite (although theoretically is only guaranteed to be positive semi-definite).

For example, $$\det [A] \geq 0$$ if $$A$$ is positive semi-definite, and $$\det [A] > 0$$ if $$A$$ is positive definite. Since the product of eigenvalues is the determinant, the implication of $$\det [A] > 0$$ is that no eigenvalue is zero. In other words, if a covariance matrix is positive definite, variability exists on all variables no matter how we transform the data.

Is this reasoning correct? Is this not almost always the case since data is never perfect? If so, are most covariance matrices actually positive definite?

• "Most" may be in the eye of the beholder. It's possible that "most" users of statistical software use redundant variables, in which case they will frequently have singular covariance matrices. – whuber Jun 2 '20 at 15:05

As @whuber said, as soon as we step into a multivariate setting, it is likely we have features that they reflect similar sources of variability. As we aggregate additional features in our analysis it becomes more likely we have collinearities and thus end up with PSD matrices.

I think the flaw in the presenting reasoning stems from your point that "variability exists on all variables no matter how we transform the data." We can have variability in all of our data features but the crux is that this variability is potentially reflecting identical information.

For example see the R code below:

set.seed(123)
x1 = runif(4)
x2 = runif(3)
A  = x1 %*% t(x2) # Obviously a rank-deficient matrix // Rank-1 matrix


While the data matrix A is of rank 1 (i.e. it has a single non-zero eigenvalue), every feature of the matrix A is having some variability.

apply(A, 2, var)
# [1] 0.08313244 0.33252977 0.74819198


The point is that all columns are perfectly correlated with each other so the covariance matrix is singular.

cor(A)
#      [,1] [,2] [,3]
# [1,]    1    1    1
# [2,]    1    1    1
# [3,]    1    1    1

• In your example, variability only exists along a single dimension because they are collinear (of course only after applying a transformation). I guess I wan't clear with my wording "no matter how we transform the data". Edit: Nevertheless, I see your point. – Ralff Jun 2 '20 at 16:23