1
$\begingroup$

I have a Gaussian process regression implementation and developed some example data to test the capabilities of those methods. In the posterior calculation one gets the covariance matrix $K$. For some sample data this matrix has a 0 determinant and thus it is not invertible. Can someone see a problem in the covariance matrix composition that leads to such behaviour?

My Covariance matrix looks like this:

$$ \begin{pmatrix} K(X,X) & K(X_*,X) \\ K(X, X_*) & K(X_*,X_*) \end{pmatrix} $$

$\endgroup$
  • $\begingroup$ It seems to be a problem with the Commons.Math Matrix LU Solver. But i guess those Implementations only react strange at some sizes of the covariance matrix. If i add one more training point everything is fine again. $\endgroup$ – Andreas Jan 11 '12 at 10:13
  • $\begingroup$ what covariance function are you using ? or you are providing a constant matrix, K as a covariance matrix ?! $\endgroup$ – user4581 Jan 11 '12 at 10:46
  • $\begingroup$ I'm using the RBF covariance function. Squared exponention. When i exclude the x* covariances from the matrix everything is fine as well. It just happens in some cases. I guess it is really a problem with the implementation of the Commons Math solver. $\endgroup$ – Andreas Jan 11 '12 at 10:56
1
$\begingroup$

A covariance matrix with zero determinant means that the random variables are perfectly correlated. If your $X$ and $X^*$ are vectors, one is an affine function of the other: $X = AX^* + B$ where $A$ is some matrix and $B$ a vector. If they are random variables, $X = aX^*+b$ where $a$ and $b$ are constants. Is there any reason to suspect that this might be happening in those cases where you are getting a zero determinant for the covariance matrix?

$\endgroup$
  • $\begingroup$ The inputs in a gaussian process are some training points and normally we set the X* points over the inputs. Thus we have more X* points than X points and they span the same range. Maybe this couls lead sometimes to the decribed X=AX* + B behavior $\endgroup$ – Andreas Jan 12 '12 at 11:49
1
$\begingroup$

for a $2 \times 2$ matrix

$\left[ \begin{array}{ll} A & B \\ C & D \end{array} \right]$

the determinant is $AD - BC$.

So in your case $K(X,X)K(X^*X^*) - K(X^*,X)K(X,X^*) = 0$.

For the RBF covariance function, $K(X,X)$ and $K(X^*X^*)$ should both be $1$, and further $0 \leq K(X^*,X) = K(X,X^*) \leq 1$. The only way to get a zero determinant is if $X = X^*$.

However I'm guessing you have more than two points ...

For block matrices, the determinant is calculated as $\det\begin{pmatrix}A& B\\ C& D\end{pmatrix} = \det(A) \det(D - C A^{-1} B)$. So maybe you can see if $A$ is invertible, calculate its determinant, and decompose this matrix.

$\endgroup$
  • $\begingroup$ The inputs in a gaussian process are some training points and normally we set the X* points over the inputs. Thus we have more X* points than X points and they span the same range. Maybe this couls lead sometimes to the decribed X=X* behavior $\endgroup$ – Andreas Jan 12 '12 at 11:49
0
$\begingroup$

Actually I the problem is in computation. In theoretical, the covariance matrix must be positive definite. However, in the computer, if you choose some points arbitrarily, it will case the determinant of covariance matrix is too small, maybe 1e-7 or more smaller than that, and now the computer thinks it is zeros, so it is not invertible.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.