4
$\begingroup$

I'm working with Gaussian process regression. Currently I start testing different covariance functions and compositions to see what type of data they could describe best. I made an own implementation in Java.

My problem: Most of the covariance functions I use result in a singular covariance matrix which is not invertible.

  1. Shouldn't the proposed covariance functions/estimators produce only invertible matrices?

  2. Are there methods or hints for regularizing the matrices? Or can that be done by using other values or ranges as inputs? May the introduction of error terms would help as well? Most problems I get with integer $x$ inputs to the Brownian motion covariance function $k(x,x') = \min(x,x')$. When I am using this the matrix it is always singular.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
7
$\begingroup$

If all covariance functions give you a singular matrix, it could be that some of your data points are identical, which gives two identical rows/columns in the matrix. To regularise the matrix, just add a ridge on the principal diagonal (as in ridge regression), which is used in Gaussian process regression as a noise term.

Note that using a composition of covariance functions or an additive combination can lead to over-fitting the marginal likelihood in evidence based model selection due to the increased number of hyper-parameters, and so can give worse results than a more basic covariance function, even though the basic covariance function is less suitable for modelling the data.

Improve this answer
$\endgroup$
4
  • $\begingroup$ I read in some publications about estimating the inverse for singular matrices. Is that a standard problem in gaussian process regression or why is there so much literature about numerical problems in the covariance matrices. $\endgroup$
    – Andreas
    Commented Jan 13, 2012 at 10:35
  • 1
    $\begingroup$ The matrix is often ill-conditioned, so there are indeed a few tricks for inverting it more reliably, there is a good discussion of this in Rasmussen and Williams book. However, I have found that I normally only run into problems when model selection tries to make a very bland RBF covariance to model an essentially linear decision boundary, so you could argue it was model mis-specification? I haven't used GP regression very much, so it is hard to know whether it crops up more often there. $\endgroup$
    – Dikran Marsupial
    Commented Jan 13, 2012 at 15:24
  • $\begingroup$ "there is a good discussion of this in Rasmussen and Williams book" - May i ask where exactly $\endgroup$
    – Andreas
    Commented Jan 16, 2012 at 9:56
  • 1
    $\begingroup$ The key trick to limiting numerical instability is on page 45 (equation 3.26). $\endgroup$
    – Dikran Marsupial
    Commented Jan 16, 2012 at 11:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.