Designing covariance matrix and kernel function for a gaussian process

Question

I am not so experienced to design a customized covariance matrix / kernel functions. I would like to get such a understanding that after looking at data, I can figure out the covariance matrix.

For example, in my case, I have a data set, $X$ that contains many zeros and couple of points far from them, close to hundred. $Y$ is like a normal distribution, $\mathcal{N}(50,10)$. $X$ and $Y$ are limited from $0$ to $100$.

So, I am try to regress $X$ on to $Y$, using the Gaussian process method. The difficulty arises from those many zeros that makes my covariance matrix messy. So, I have a large standard deviation for whole estimation.

Answer 1

You have to choose the functional form of the covariance based on your prior knowledge of the smoothness of the data. Usual covariance functions are like $$w(x,y) = \tau^2\exp\left(-\frac{1}{2}\left(\frac{x-y}{\lambda}\right)^\alpha\right)$$ and it is very common to choose $\alpha=2$ because it leads to a continuous and differentiable function. If you choose $\alpha\le1$ the resulting function is continuous but not smooth. See MacKay, page 545 or the whole chapter.

For the other parameters ($\tau$,$\lambda$, and a noise term $\sigma$ if you add it) you have to optimize them to maximize the marginal likelihood. See Rasmussen and Williams chapter 4 which will help you to understand everything in more detail.

Answer 2

I've not worked through the details of GPs, so I cannot help you there.

However, it seems like you have two groups of data in X: X=0 and X>0. You may get better results by first classifying X into these two groups based on Y, and then performing GP regression in the X>0 class.