Tell me more ×
Cross Validated is a question and answer site for statisticians, data analysts, data miners and data visualization experts. It's 100% free, no registration required.
up vote 0 down vote favorite

I'm working with gaussian process regression. Currently I start testing differnt covariance functions and compositions to see what type of data they could describe best. I made an own implementation in Java.

My problem: Most of the covariance funtions I use result in a singular covariance matrix which is not invertable.

  1. Should't the proposed covariance functions/estimators produce only invertable matrices?

  2. Are there methods or hints for regularizing the matrices? Or can that be done by using other values or ranges as inputs? May the introduction of error terms would help as well? Most problems I get with integer $x$ inputs to the brownian motion covariance function $k(x,x') = \min(x,x')$. When I am using this the matrix it is always singular.

Thank you

share|improve this question

1 Answer

up vote 3 down vote accepted

If all covariance functions give you a singular matrix, it could be that some of your data points are identical, which gives two identical rows/columns in the matrix. To regularise the matrix, just add a ridge on the principal diagonal (as in ridge regression), which is used in Gaussian process regression as a noise term.

Note that using a composition of covariance functions or an additive combination can lead to over-fitting the marginal likelihood in evidence based model selection due to the increased number of hyper-parameters, and so can give worse results than a more basic covariance function, even though the basic covariance function is less suitable for modelling the data.

share|improve this answer
I read in some publications about estimating the inverse for singular matrices. Is that a standard problem in gaussian process regression or why is there so much literature about numerical problems in the covariance matrices. – Andreas Jan 13 '12 at 10:35
1  
The matrix is often ill-conditioned, so there are indeed a few tricks for inverting it more reliably, there is a good discussion of this in Rasmussen and Williams book. However, I have found that I normally only run into problems when model selection tries to make a very bland RBF covariance to model an essentially linear decision boundary, so you could argue it was model mis-specification? I haven't used GP regression very much, so it is hard to know whether it crops up more often there. – Dikran Marsupial Jan 13 '12 at 15:24
"there is a good discussion of this in Rasmussen and Williams book" - May i ask where exactly – Andreas Jan 16 '12 at 9:56
1  
The key trick to limiting numerical instability is on page 45 (equation 3.26). – Dikran Marsupial Jan 16 '12 at 11:04

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.