I'm working with gaussian process regression. Currently I start testing differnt covariance functions and compositions to see what type of data they could describe best. I made an own implementation in Java.
My problem: Most of the covariance funtions I use result in a singular covariance matrix which is not invertable.
Should't the proposed covariance functions/estimators produce only invertable matrices?
Are there methods or hints for regularizing the matrices? Or can that be done by using other values or ranges as inputs? May the introduction of error terms would help as well? Most problems I get with integer $x$ inputs to the brownian motion covariance function $k(x,x') = \min(x,x')$. When I am using this the matrix it is always singular.