# Gaussian Process covariance matrix gets zero determinant

I have a Gaussian process regression implementation and developed some example data to test the capabilities of those methods. In the posterior calculation one gets the covariance matrix $K$. For some sample data this matrix has a 0 determinant and thus it is not invertible. Can someone see a problem in the covariance matrix composition that leads to such behaviour?

My Covariance matrix looks like this:

$$\begin{pmatrix} K(X,X) & K(X_*,X) \\ K(X, X_*) & K(X_*,X_*) \end{pmatrix}$$

-
It seems to be a problem with the Commons.Math Matrix LU Solver. But i guess those Implementations only react strange at some sizes of the covariance matrix. If i add one more training point everything is fine again. –  Andreas Jan 11 '12 at 10:13
what covariance function are you using ? or you are providing a constant matrix, K as a covariance matrix ?! –  user4581 Jan 11 '12 at 10:46
I'm using the RBF covariance function. Squared exponention. When i exclude the x* covariances from the matrix everything is fine as well. It just happens in some cases. I guess it is really a problem with the implementation of the Commons Math solver. –  Andreas Jan 11 '12 at 10:56

for a $2 \times 2$ matrix

$\left| \begin{array}{ll} A & B \\ C & D \end{array} \right|$

the determinant is $AB - CD$.

So in your case $K(X,X)K(X^*X^*) - K(X^*,X)K(X,X^*) = 0$.

For the RBF covariance function, $K(X,X)$ and $K(X^*X^*)$ should both be $1$, and further $0 \leq K(X^*,X) = K(X,X^*) \leq 1$. The only way to get a zero determinant is if $X = X^*$.

However I'm guessing you have more than two points ...

For block matrices, the determinant is calculated as $\det\begin{pmatrix}A& B\\ C& D\end{pmatrix} = \det(A) \det(D - C A^{-1} B)$. So maybe you can see if $A$ is invertible, calculate its determinant, and decompose this matrix.

-
The inputs in a gaussian process are some training points and normally we set the X* points over the inputs. Thus we have more X* points than X points and they span the same range. Maybe this couls lead sometimes to the decribed X=X* behavior –  Andreas Jan 12 '12 at 11:49
A covariance matrix with zero determinant means that the random variables are perfectly correlated. If your $X$ and $X^*$ are vectors, one is an affine function of the other: $X = AX^* + B$ where $A$ is some matrix and $B$ a vector. If they are random variables, $X = aX^*+b$ where $a$ and $b$ are constants. Is there any reason to suspect that this might be happening in those cases where you are getting a zero determinant for the covariance matrix?