Designing covariance matrix and kernel function for a gaussian process

Question

I am not so experience to design a customized covariance matrix / kernel functions. I would like to get such a feeling that after looking data, figure out the covariance matrix.

For example, in my case, I have a dataset, $X$ contains many zeros and couple of points far from them close to hundred. $Y$ is like a normal distribution, $N(50,10)$. $X$ and $Y$ are limited from 0 to 100.

So, I am try to regress $X$ on $Y$, using gaussian process method. The difficulty arise from those many zeros that makes my covariant matrix messy. So, I have a large standard deviation for whole estimation.

Answer 1

You have to choose the functional form of the covariance based on your prior knowledge of the smoothness of the data. Usual covariance functions are like $$w(x,y) = \tau^2\exp\left(-\frac{1}{2}\left(\frac{x-y}{\lambda}\right)^\alpha\right)$$ and it is very common to choose $\alpha=2$ because it leads to a continuous and differentiable function. If you choose $\alpha\le1$ the resulting function is continuous but not smooth. See MacKay, page 545 or the whole chapter.

For the other parameters ($\tau$,$\lambda$, and a noise term $\sigma$ if you add it) you have to optimize them to maximize the marginal likelihood. See Rasmussen and Williams chapter 4 which will help you to understand everything in more detail.

Answer 2

I've not worked through the details of GPs, so I cannot help you there.

However, it seems like you have two groups of data in X: X=0 and X>0. You may get better results by first classifying X into these two groups based on Y, and then performing GP regression in the X>0 class.