I am currently playing around with Gaussian process regression. I discovered some confusing facts that I would like to be answered:

  1. When I optimize the parameters like the length and error terms for the covariance function I get different results for scaled targets like $y \times 100$. This leads to the next thing:

  2. When I use very small targets between $0$ and $0.1$ the whole process seems to perform very poor.

Can somebody explain these behaviors to me or maybe can send a link where those problems are discussed?


1 Answer 1


The hyper-parameters should be expected to change somewhat, if only because scaling the target will also scale the noise variance. Maximising the evidence is not really a proper Bayesian thing to do, ideally you should place an appropriate prior over the hyper-parameters and integrate them out as well, however in practice most people don't do this due to the computational expense involved. I suspect if you integrated the hyper-parameters out rather than maximising the evidence, then both models would give similar performance.

The trouble with poor performance with a small scaling factor may be due to the starting point for the hyper-parameter search being a long way from the optimal value, on a plateau, or worse near a local minima. Changing the scale will change the noise variance, which might be enough to move the initial hyper-parameter values to somewhere nearer the optimal solution. Try maximising the evidence starting from different (random?) initial points and see whether they call converge to the same solution.

Model selection for GPs is not straightforward, especially by evidence maximisation. The evidence is noisy (it is based on a small sample of data, so small changes in the particular sample used can give susbtantial changes in the evidence) and may well have local minima etc.

Caveat: I haven't used Gaussian process regression much, but most of the comments above stem from my experiments with Gaussian Process CLassification.

  • $\begingroup$ Thank you for the answer. I forgot to optimize sigma_n. I only optimized sigma_f. This caused the strange behavior. But your displayed thoughts are very useful. Thanks $\endgroup$
    – Andreas
    Jan 10, 2012 at 14:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.