1
$\begingroup$

Context: I'm working on a machine learning problem where I'm using multivariate normal likelihood which requires calculating determinant and inverting the covariance matrix. I'm trying to generate some simulation data to prove that the method works but without success.

Question: Under what condition is a sample covariance matrix invertible? To illustrate, consider the below Python code which generates: 1) 3 random walks; 2) random walks around a trend. (1) returns determinant of zero and (2) returns a non-zero determinant. So what condition must be satisfied in order to generate a covariance matrix that's invertible?

import numpy as np

# creates 3 random walks
y = np.cumsum(np.random.normal(0., 0.1, (50, 3)), axis=0)

# calculate covariance matrix
np.cov(y, rowvar=False)
c = np.cov(y, rowvar=False)

# determinant is basically zero
print(np.linalg.det(c))


# add a linear trend to the 3 random walks
y = y + np.atleast_2d(np.arange(50)).T
c = np.cov(y, rowvar=False)

# prints 0.2218351316148853
np.linalg.det(c)
$\endgroup$

1 Answer 1

1
$\begingroup$

In Deep Learning by Goodfellow et al, the authors write on page 232:

[$\mathbf{X^\top X}$] can be singular whenever the data-generating distribution truly has no variance in some direction, or when no variance is observed in some direction because there are fewer examples (rows of $\mathbf{X}$) than input features (columns of $\mathbf{X}$).

This questiton/answer, from which the previous paragraph was taken, can be useful.

Sometimes, the sample covariance matrix will be singular and you have to face it. There are approaches to handle such situations, as can be seen here, with $\ell_1$ regularization.

$\endgroup$
2
  • 1
    $\begingroup$ Thanks. It turns out that the way I was estimating the covariance matrix meant that it will more often than not be non-invertible because, as you've pointed out, I don't have enough rows to estimate $XX^T$ (I was trying to estimate a $N \times N$ covariance matrix with just $N$ samples. $\endgroup$
    – stevew
    Dec 17, 2021 at 3:53
  • $\begingroup$ If my answer answered your question, please consider accepting it as an answer and upvoting 😁 $\endgroup$ Dec 17, 2021 at 9:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.