I am not so experience to design a customized covariance matrix / kernel functions. I would like to get such a feeling that after looking data, figure out the covariance matrix.
For example, in my case, I have a dataset, $X$ contains many zeros and couple of points far from them close to hundred. $Y$ is like a normal distribution, $N(50,10)$. $X$ and $Y$ are limited from 0 to 100.
So, I am try to regress $X$ on $Y$, using gaussian process method. The difficulty arise from those many zeros that makes my covariant matrix messy. So, I have a large standard deviation for whole estimation.