# Gaussian Process covariance matrix gets zero determinant

i have a gaussian process regression implementation and developed some example data to test the capabilities of those methods. In the posterior calculation one gets the covariance matrix K. For some sample data this matrix gets a 0 determinant and thus it is not invertable. Can someone see a problem in the covariance matrix composition that lead to such behavior?

My Covariance matrix looks line {K(X,X) K(X*,X) ; K(X, X*) K(X*,X*)}

-
 It seems to be a problem with the Commons.Math Matrix LU Solver. But i guess those Implementations only react strange at some sizes of the covariance matrix. If i add one more training point everything is fine again. – Andreas Jan 11 '12 at 10:13 what covariance function are you using ? or you are providing a constant matrix, K as a covariance matrix ?! – user4581 Jan 11 '12 at 10:46 I'm using the RBF covariance function. Squared exponention. When i exclude the x* covariances from the matrix everything is fine as well. It just happens in some cases. I guess it is really a problem with the implementation of the Commons Math solver. – Andreas Jan 11 '12 at 10:56

for a $2 \times 2$ matrix

$\left| \begin{array}{ll} A & B \\ C & D \end{array} \right|$

the determinant is $AB - CD$.

So in your case $K(X,X)K(X^*X^*) - K(X^*,X)K(X,X^*) = 0$.

For the RBF covariance function, $K(X,X)$ and $K(X^*X^*)$ should both be $1$, and further $0 \leq K(X^*,X) = K(X,X^*) \leq 1$. The only way to get a zero determinant is if $X = X^*$.

However I'm guessing you have more than two points ...

For block matrices, the determinant is calculated as $\det\begin{pmatrix}A& B\\ C& D\end{pmatrix} = \det(A) \det(D - C A^{-1} B)$. So maybe you can see if $A$ is invertible, calculate its determinant, and decompose this matrix.

-
 The inputs in a gaussian process are some training points and normally we set the X* points over the inputs. Thus we have more X* points than X points and they span the same range. Maybe this couls lead sometimes to the decribed X=X* behavior – Andreas Jan 12 '12 at 11:49

A covariance matrix with zero determinant means that the random variables are perfectly correlated. If your $X$ and $X^*$ are vectors, one is an affine function of the other: $X = AX^* + B$ where $A$ is some matrix and $B$ a vector. If they are random variables, $X = aX^*+b$ where $a$ and $b$ are constants. Is there any reason to suspect that this might be happening in those cases where you are getting a zero determinant for the covariance matrix?

-
 The inputs in a gaussian process are some training points and normally we set the X* points over the inputs. Thus we have more X* points than X points and they span the same range. Maybe this couls lead sometimes to the decribed X=AX* + B behavior – Andreas Jan 12 '12 at 11:49