I am trying to perform Gaussian Process for regression. I chose the SE Kernel : $K(x_i,x_j) = \exp(-\frac{||x_i-x_j||^2}{l}) + \sigma_n\delta_{i,j}$. I begin by maximize the log-likelihood with respect to the latent variables, with conservative values for hyperparameters.

Then I maximize the log-likelihood with respect to the hyperparameters and the results seems weird : the parameter $\sigma_n$ does not stop increasing. I don't get why the log-likelihood grows always with $\sigma_n$ while the model obviously goes wrong.

I tried also to maximize the log-likelihood only with respect to the length-scale parameter $l$. In this case, the log-likelihood is the highest when $l$ equal the value used to perform the first optimization with respect to the hyperparameters.

I both perform the optimization of the log-likelihood with respect to the latent variables by maximizing

$-\frac{dn}{2} - \frac{d}{2}\ln(|K|) - \frac{1}{2}\text{tr}(K^{-1}YY^T)$

where $Y$ is the $n \times d$ matrix of observed data in the function-space. Is there something I poorly undesrtood ?


1 Answer 1


If you check out the GPML book there is a natural trade off between model fit and complexity as expressed in the determinant (complexity of model) and the trace term above (data fit). As the noise term gets big the data fit is going to look really good but the complexity is increasingly worse. My guess is that you are getting trapped in a local minima. You could try optimisation with multiple restarts or bound the space of optimisation. You can also put a prior over the hyperparameter space.

Also the log marginal likelihood seen in texts is written as,

$-\frac{1}{2}y^tK^{-1}y - \frac{1}{2}log(|K|) - \frac{d}{2}log(2\pi)$

  • $\begingroup$ Thank you ! Indeed the log-likelihood as i wrote in my thread was wrong. In my case i have $n$ observed points in a $d$-dimensionnal space, that's why i have $-\frac{1}{2}\text{tr}(K^{-1} Y Y^T)$ instead (cf jmlr.org/papers/volume6/lawrence05a/lawrence05a.pdf eq6) About the local minima : do you suggest me to optimize the log-likelihood with respect to the hyperparameters using another set of latent variables than those found by the first maximization ? Last question : how do one put a prior on hyperparameters ? i read rasmussen's book but i am strunggling to understand it by myself. $\endgroup$
    – Irminsul
    Sep 25, 2015 at 11:39
  • $\begingroup$ Ah Neil Lawrence is one of the gods of this domain alright so you picked a good paper to cite! I haven't done a huge amount of work with the LVM GPs but I would imagine you should be optimising hyperparameters as your latent variables change. Put a prior on the hyperparameter space by multiplying the likelihood by how probable the the hyperparameters are - P(GP | hyp.)P(hyp.), basically adding the log probability of the hyperparameters to the log marginal likelihood term. Neil has the GP LVM code built into GPy if you are using python or want the code as a reference - check out their Github :) $\endgroup$
    – j__
    Sep 25, 2015 at 11:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.