15
$\begingroup$

Log marginal likelihood for Gaussian Process as per Rasmussen's Gaussian Processes for Machine Learning equation 2.30 is:

$$\log p(y|X) = -\frac{1}{2}y^T(K+\sigma^2_n I)^{-1}y - \frac{1}{2}\log|K+\sigma^2_n I|-\frac{n}{2}\log2\pi$$

Where as Matlab's documentation on Gaussian Process formulates the relation as

$$\log p(y|X, \beta, \theta, \sigma^2) = -\frac{1}{2}\left(y-H\beta\right)^T(K+\sigma^2_n I)^{-1}\left(y-H\beta\right) - \frac{1}{2}\log|K+\sigma^2_n I|-\frac{n}{2}\log2\pi$$

where $H$ is the vector of basis functions and $\beta$ is coefficient vector.

My doubts:

  1. Why there is difference in the two relations?
  2. From my understanding, $H\beta$ is prediction from Gaussian Process; am I right?

Thanks

Improve this question
$\endgroup$
1
  • $\begingroup$ I'm trying to see where this log marginal likelihood came from, but I'm struggling to understand it. I've flipped through the resource you've linked; seems like something I'll need to accept without supporting intuition of how it was derived. $\endgroup$
    – jbuddy_13
    Jan 1, 2021 at 21:04

1 Answer 1

Reset to default
10
$\begingroup$

The more general formulation for the log marginal likelihood (not marginal log likelihood, as you originally wrote - I edited it in your post) of a GP is

$$\log p(y|X) = -\frac{1}{2}(y - m(X))^T(K+\sigma^2_n I)^{-1}(y - m(X)) - \frac{1}{2}\log|K+\sigma^2_n I|-\frac{n}{2}\log2\pi$$

where $m(x): \mathbb{R}^d \rightarrow \mathbb{R}$ for a given point $x$ is a mean function of a GP; and the notation $m(X)$ represents a vector function obtained by applying the mean function to every point in $X$. The GP in GPML (Eq. 2.30) is a zero-mean GP.

In the Matlab version, $H \beta$ stands for a mean function expressed as a linear combination of basis functions $H = H(x)$, it is not the prediction of the GP.

The GP mean prediction will revert to the mean function very far away from points in the training set $X$ (very far in terms of length scale of the kernel), but it is going to be generally different otherwise.

Improve this answer
$\endgroup$
6
  • $\begingroup$ thanks @lacerbi, $m(X)$ is $ = K(X_*, X)[K(X,X)+\sigma^2\mathrm{I}]^{-1}y$ for a zero mean process, right?? $\endgroup$
    – pkj
    May 17, 2017 at 10:55
  • $\begingroup$ Matlab uses the term marginal log likelihood $\endgroup$
    – pkj
    May 17, 2017 at 11:00
  • $\begingroup$ I think you may be confusing the (a priori) mean function $m(X)$ with the (posterior) mean prediction, sometimes written as $\mu(X|\theta, \sigma, X, y)$ or $\bar{f}_*$. What you are writing is the GP mean prediction, and it is correct in that sense (see Eq. 2.25 in the GPML book). And Matlab is wrong then, it is log marginal likelihood. $\endgroup$
    – lacerbi
    May 17, 2017 at 11:02
  • $\begingroup$ sorry @lacerbi, but I don't know what a priori mean function is. Do you mean to say $m(X)$ is as in the relation $f(X_*) = m(X_*) +K(X_*, X)[K(X,X)+\sigma^2\mathrm{I}]^{-1}(y-m(X))$ for a non-zero mean process $\endgroup$
    – pkj
    May 17, 2017 at 11:46
  • 1
    $\begingroup$ Yep. By the way, by a priori I meant that that's what you would expect from the GP if you had no training points (or far from them). In practice, the mean function is parametrized, and we fit these parameters as part of the GP model. Often the mean function is chosen to be a constant, but more in general you can have whatever you want, such as a linear combination of aptly chosen basis functions. In practice, I haven't seen it used much, since usually you would let the GP pick up features by itself (via the kernel). $\endgroup$
    – lacerbi
    May 17, 2017 at 12:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.