Log marginal likelihood for Gaussian Process as per Rasmussen's Gaussian Processes for Machine Learning equation 2.30 is:

$$\log p(y|X) = -\frac{1}{2}y^T(K+\sigma^2_n I)^{-1}y - \frac{1}{2}\log|K+\sigma^2_n I|-\frac{n}{2}\log2\pi$$

Where as Matlab's documentation on Gaussian Process formulates the relation as

$$\log p(y|X, \beta, \theta, \sigma^2) = -\frac{1}{2}\left(y-H\beta\right)^T(K+\sigma^2_n I)^{-1}\left(y-H\beta\right) - \frac{1}{2}\log|K+\sigma^2_n I|-\frac{n}{2}\log2\pi$$

where $H$ is the vector of basis functions and $\beta$ is coefficient vector.

My doubts:

  1. Why there is difference in the two relations?
  2. From my understanding, $H\beta$ is prediction from Gaussian Process; am I right?

Thanks

up vote 1 down vote accepted

The more general formulation for the log marginal likelihood (not marginal log likelihood, as you originally wrote - I edited it in your post) of a GP is

$$\log p(y|X) = -\frac{1}{2}(y - m(X))^T(K+\sigma^2_n I)^{-1}(y - m(X)) - \frac{1}{2}\log|K+\sigma^2_n I|-\frac{n}{2}\log2\pi$$

where $m(x): \mathbb{R}^d \rightarrow \mathbb{R}$ for a given point $x$ is a mean function of a GP; and the notation $m(X)$ represents a vector function obtained by applying the mean function to every point in $X$. The GP in GPML (Eq. 2.30) is a zero-mean GP.

In the Matlab version, $H \beta$ stands for a mean function expressed as a linear combination of basis functions $H = H(x)$, it is not the prediction of the GP.

The GP mean prediction will revert to the mean function very far away from points in the training set $X$ (very far in terms of length scale of the kernel), but it is going to be generally different otherwise.

  • thanks @lacerbi, $m(X)$ is $ = K(X_*, X)[K(X,X)+\sigma^2\mathrm{I}]^{-1}y$ for a zero mean process, right?? – pkj May 17 '17 at 10:55
  • Matlab uses the term marginal log likelihood – pkj May 17 '17 at 11:00
  • I think you may be confusing the (a priori) mean function $m(X)$ with the (posterior) mean prediction, sometimes written as $\mu(X|\theta, \sigma, X, y)$ or $\bar{f}_*$. What you are writing is the GP mean prediction, and it is correct in that sense (see Eq. 2.25 in the GPML book). And Matlab is wrong then, it is log marginal likelihood. – lacerbi May 17 '17 at 11:02
  • sorry @lacerbi, but I don't know what a priori mean function is. Do you mean to say $m(X)$ is as in the relation $f(X_*) = m(X_*) +K(X_*, X)[K(X,X)+\sigma^2\mathrm{I}]^{-1}(y-m(X))$ for a non-zero mean process – pkj May 17 '17 at 11:46
  • Yep. By the way, by a priori I meant that that's what you would expect from the GP if you had no training points (or far from them). In practice, the mean function is parametrized, and we fit these parameters as part of the GP model. Often the mean function is chosen to be a constant, but more in general you can have whatever you want, such as a linear combination of aptly chosen basis functions. In practice, I haven't seen it used much, since usually you would let the GP pick up features by itself (via the kernel). – lacerbi May 17 '17 at 12:07

Your Answer

 
discard

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.