I'm working with Gaussian process regression. Currently I start testing different covariance functions and compositions to see what type of data they could describe best. I made an own implementation in Java.
My problem: Most of the covariance functions I use result in a singular covariance matrix which is not invertible.
Shouldn't the proposed covariance functions/estimators produce only invertible matrices?
Are there methods or hints for regularizing the matrices? Or can that be done by using other values or ranges as inputs? May the introduction of error terms would help as well? Most problems I get with integer $x$ inputs to the Brownian motion covariance function $k(x,x') = \min(x,x')$. When I am using this the matrix it is always singular.