I am not so experienced to design a customized covariance matrix / kernel functions. I would like to get such a understanding that after looking at data, I can figure out the covariance matrix.

For example, in my case, I have a data set, $X$ that contains many zeros and couple of points far from them, close to hundred. $Y$ is like a normal distribution, $\mathcal{N}(50,10)$. $X$ and $Y$ are limited from $0$ to $100$.

So, I am try to regress $X$ on to $Y$, using the Gaussian process method. The difficulty arises from those many zeros that makes my covariance matrix messy. So, I have a large standard deviation for whole estimation.

  • $\begingroup$ Cleared a bit; I hope you didn't meant some other GP method. $\endgroup$
    – user88
    May 12, 2011 at 23:38

2 Answers 2


You have to choose the functional form of the covariance based on your prior knowledge of the smoothness of the data. Usual covariance functions are like $$w(x,y) = \tau^2\exp\left(-\frac{1}{2}\left(\frac{x-y}{\lambda}\right)^\alpha\right)$$ and it is very common to choose $\alpha=2$ because it leads to a continuous and differentiable function. If you choose $\alpha\le1$ the resulting function is continuous but not smooth. See MacKay, page 545 or the whole chapter.

For the other parameters ($\tau$,$\lambda$, and a noise term $\sigma$ if you add it) you have to optimize them to maximize the marginal likelihood. See Rasmussen and Williams chapter 4 which will help you to understand everything in more detail.


I've not worked through the details of GPs, so I cannot help you there.

However, it seems like you have two groups of data in X: X=0 and X>0. You may get better results by first classifying X into these two groups based on Y, and then performing GP regression in the X>0 class.

  • $\begingroup$ O yes, I was thinking same, sort of putting into a mixture model. But still, I am wondering how can I choose a proper covariance matrix and set its hyper-parameter. I am playing with a code, and changing parameter without having an idea is embarrassing ! $\endgroup$
    – Areza
    May 13, 2011 at 17:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.