# Hyperparameter estimation in Gaussian process

I am trying to optimize the hyperparameters for a Gaussian process. As a starter I choose the squared exponential function for covariance where iI have to optimize 3 parameters $\sigma_f$, $\sigma_n$ and the length parameter $l$.

$$k_y(x_p,x_q) = \sigma^2_f \exp\left(-\frac{1}{2l^2}(x_p-x_q)^2\right) + \sigma^2_n\delta_{pq}$$

As described in  I try to maximize the marginal likelihood using the parameters. To achieve this I use Commons Math optimizers and compute the gradients of the marginal likelihood function.

Everything works fine except for a tiny example if I only use $\sigma_f$ and $l$ for the optimization. As soon as I try to optimize $\sigma_n$ as well, I run into numerical problems and the the covariance matrix of the process is NOT positive-definite anymore after a few iterations of the optimizer and thus produces INFINITY or NAN errors within the optimizer.

Can anyone explain that behavior and the connection to the hyperparameter $\sigma_n$?

 Gaussian Processes for Machine Learning Carl Edward Rasmussen and Christopher K. I. Williams The MIT Press, 2006. ISBN 0-262-18253-X.

• What do you mean by "tiny example"? Small sample size? Jun 8, 2012 at 13:26
• Yes i just evaluate the example: y = {0.55*-3.0, 0.55*-2.0, 0.55*-0.6, 0.55*0.4, 0.55*1.0, 0.55*1.6} X = {-1.5, -1.0, -.75, -0.4, -0.25 ,0}; Jun 8, 2012 at 13:41
• Could you try this again with a sample of size 12? And add a little fuzz to the x-values, so they aren't quite so close to being on a grid... I'm thinking your small sample size might, but only might, be the problem. Jun 8, 2012 at 17:41
• I had the Ininity/NAN problem as well for a long time until I switched to using SVD instead of the native cholesky that GPML use. The problem is that with a smooth kernel like this the singular values die down pretty quickly. SVD can alleviate problem to some level if you ignore eigen values below a certain threshold. Nov 15, 2013 at 4:25
• It is very common to get non semidefinite covariance matrix due to numerical issue. you then need to increase the lower bound of noise level for your optimization. Dec 2, 2016 at 12:34

I suspect what is happening is that the model has decided that an essentially linear classifier would be best, and has set $\sigma_n$ to zero, as the noise component is not seen as necessary (which is a shame as it adds a ridge to the covariance matrix, which helps ensure that it is p.d.). In that case, only the SEiso bit is used, so $\sigma_f$ will probably be much larger than $\sigma_n$, however to make a linear classifier it will try and make $l$ as large as possible, which seems to end up resulting in numerical problems when evaluating the bit inside the exponential. I'm a pretty heavy user of GPML and have seen this a fair bit. One solution is to limit the magnitudes of the logarithms of the hyper-parameters during the search (which is equivalent to having a hyper-prior on the hyper-parameters), which tends to prevent this from happening. If you print out the values of the hyper-parameters, then the last ones that get printed before it goes "bang" will give you a good idea where to place the limits. This tends not to affect performance very much, the generalisation error at such points in hyper-parameter tends to be fairly flat, which causes gradient descent methods to take large steps that put you far enough from the origin that you get numerical accuracy problems.