# Gaussian Process covariance matrix gets zero determinant

I have a Gaussian process regression implementation and developed some example data to test the capabilities of those methods. In the posterior calculation one gets the covariance matrix $K$. For some sample data this matrix has a 0 determinant and thus it is not invertible. Can someone see a problem in the covariance matrix composition that leads to such behaviour?

My Covariance matrix looks like this:

$$\begin{pmatrix} K(X,X) & K(X_*,X) \\ K(X, X_*) & K(X_*,X_*) \end{pmatrix}$$

• It seems to be a problem with the Commons.Math Matrix LU Solver. But i guess those Implementations only react strange at some sizes of the covariance matrix. If i add one more training point everything is fine again. Jan 11, 2012 at 10:13
• what covariance function are you using ? or you are providing a constant matrix, K as a covariance matrix ?! Jan 11, 2012 at 10:46
• I'm using the RBF covariance function. Squared exponention. When i exclude the x* covariances from the matrix everything is fine as well. It just happens in some cases. I guess it is really a problem with the implementation of the Commons Math solver. Jan 11, 2012 at 10:56

A covariance matrix with zero determinant means that the random variables are perfectly correlated. If your $X$ and $X^*$ are vectors, one is an affine function of the other: $X = AX^* + B$ where $A$ is some matrix and $B$ a vector. If they are random variables, $X = aX^*+b$ where $a$ and $b$ are constants. Is there any reason to suspect that this might be happening in those cases where you are getting a zero determinant for the covariance matrix?

• The inputs in a gaussian process are some training points and normally we set the X* points over the inputs. Thus we have more X* points than X points and they span the same range. Maybe this couls lead sometimes to the decribed X=AX* + B behavior Jan 12, 2012 at 11:49
• your answer has good intuition. I am also interested to know why do we then minimize the determinant of log-likelihood of the Gaussian process to determine the hyperparameters of the kernel function? Is the intuition that if the determinant of log likelihood is 0 then we have a perfect hyperparameters? Apr 7, 2020 at 21:00

for a $2 \times 2$ matrix

$\left[ \begin{array}{ll} A & B \\ C & D \end{array} \right]$

the determinant is $AD - BC$.

So in your case $K(X,X)K(X^*X^*) - K(X^*,X)K(X,X^*) = 0$.

For the RBF covariance function, $K(X,X)$ and $K(X^*X^*)$ should both be $1$, and further $0 \leq K(X^*,X) = K(X,X^*) \leq 1$. The only way to get a zero determinant is if $X = X^*$.

However I'm guessing you have more than two points ...

For block matrices, the determinant is calculated as $\det\begin{pmatrix}A& B\\ C& D\end{pmatrix} = \det(A) \det(D - C A^{-1} B)$. So maybe you can see if $A$ is invertible, calculate its determinant, and decompose this matrix.

• The inputs in a gaussian process are some training points and normally we set the X* points over the inputs. Thus we have more X* points than X points and they span the same range. Maybe this couls lead sometimes to the decribed X=X* behavior Jan 12, 2012 at 11:49

Actually I the problem is in computation. In theoretical, the covariance matrix must be positive definite. However, in the computer, if you choose some points arbitrarily, it will case the determinant of covariance matrix is too small, maybe 1e-7 or more smaller than that, and now the computer thinks it is zeros, so it is not invertible.

• Jun 21, 2016 at 15:47
• yeah, you should make sure that Jan 17, 2017 at 17:57