log marginal likelihood for Gaussian Process as given by Rasmussen's: Gaussian Processes for Machine Learning equation 5.8 is $$\log p(y|X, \theta) = -\frac{1}{2}y^{T} K_y^{-1}y - \frac{1}{2}\log|K_y|-\frac{n}{2}\log2\pi$$ where $K_y = K_f + \sigma^2I$ is the covariance matrix for the noisy targets $y$ (and $K_f$ is the covariance matrix for the noise-free latent $f$) and $\theta$ is hyperparameters (parameters of kernel function).

Partial derivatives of the marginal likelihood w.r.t. the hyperparameters is given in eqn. 5.9.

$$\frac{\partial}{\partial\theta_i} \mathrm{log}P(y|x, \theta) = \frac{1}{2}y^TK^{-1}\frac{\partial K}{\partial\theta_i}K^{-1}y^T -\frac{1}{2}\mathrm{tr}\big(K^{-1}\frac{\partial K}{\partial\theta_i}\big) $$ Now my doubt is what is $K$ in equation 5.9, does it represents $K_y$ or $K_f$ or something else.

  • 1
    $\begingroup$ It represents $K_y$. You are just differentiating the first Eq. $\endgroup$ – lacerbi May 18 '17 at 9:03
  • $\begingroup$ Can I also differentiate and optimize for $\sigma$ $\endgroup$ – pkj May 18 '17 at 9:23
  • 1
    $\begingroup$ Yes, you can optimize wrt $\sigma$, but in some cases the solution is not good, as you need also to control numerical quality of the problem via $\sigma$. $\endgroup$ – Alexey Zaytsev May 18 '17 at 9:34
  • $\begingroup$ then I would optimize $\sigma$ using MCMC, any other suggestion for estimating $\sigma$ $\endgroup$ – pkj May 18 '17 at 9:52
  • $\begingroup$ Look up the GPML code. gaussianprocess.org/gpml/code/matlab/doc It's possibly also discussed in the book. You might want to do different computations depending on wheter $\sigma$ is large or small, in order to maintain numerical precision. $\endgroup$ – lacerbi May 18 '17 at 16:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.