I have a Gaussian process regression implementation and developed some example data to test the capabilities of those methods. In the posterior calculation one gets the covariance matrix $K$. For some sample data this matrix has a 0 determinant and thus it is not invertible. Can someone see a problem in the covariance matrix composition that leads to such behaviour?
My Covariance matrix looks like this:
$$ \begin{pmatrix} K(X,X) & K(X_*,X) \\ K(X, X_*) & K(X_*,X_*) \end{pmatrix} $$
A covariance matrix with zero determinant means that the random variables are perfectly correlated. If your $X$ and $X^*$ are vectors, one is an affine function of the other: $X = AX^* + B$ where $A$ is some matrix and $B$ a vector. If they are random variables, $X = aX^*+b$ where $a$ and $b$ are constants. Is there any reason to suspect that this might be happening in those cases where you are getting a zero determinant for the covariance matrix?
for a $2 \times 2$ matrix
$\left[ \begin{array}{ll} A & B \\ C & D \end{array} \right]$
the determinant is $AD - BC$.
So in your case $K(X,X)K(X^*X^*) - K(X^*,X)K(X,X^*) = 0$.
For the RBF covariance function, $K(X,X)$ and $K(X^*X^*)$ should both be $1$, and further $0 \leq K(X^*,X) = K(X,X^*) \leq 1$. The only way to get a zero determinant is if $X = X^*$.
However I'm guessing you have more than two points ...
For block matrices, the determinant is calculated as $\det\begin{pmatrix}A& B\\ C& D\end{pmatrix} = \det(A) \det(D - C A^{-1} B)$. So maybe you can see if $A$ is invertible, calculate its determinant, and decompose this matrix.
Actually I the problem is in computation. In theoretical, the covariance matrix must be positive definite. However, in the computer, if you choose some points arbitrarily, it will case the determinant of covariance matrix is too small, maybe 1e-7 or more smaller than that, and now the computer thinks it is zeros, so it is not invertible.
