1
$\begingroup$

The log marginal likelihood for Gaussian Process regression is calculated according to Chapter 5 of the Rasmussen and Williams GPML book:

$log\ p(y|X,\theta) = -\frac{1}{2}y^TK_y^{-1}y-\frac{1}{2}log|K_y|-\frac{n}{2}log2\pi$

It is straightforward to get a single log marginal likelihood value when the regression output is one dimension. But when it comes to multiple-output regression, the first term on the right side of the above equation is actually a matrix instead of a single value:

$-\frac{1}{2}y^TK_y^{-1}y$

So I am wondering how to handle this situation. I notice in many Gaussian Process implementations, people get the log marginal likelihood by summing the posterior pdf of each training sample, which is similar to the "LOO-CV" concept mentioned in Chapter 5 of the GPML book, but without really leaving one out during training. Is this a reasonable way to calculate the log marginal likelihood in Gaussian Process context? Thanks for any feedback in advance!

$\endgroup$
2
$\begingroup$

Terms like $y^tMy$ are scalar. It's known as a quadratic form. In your case, $M=K_y^{-1}$.

I assume that $K$ is a function of $\theta$, the vector of unknown parameters. You grab the partial derivatives with respect to the $\theta$ vector and solve for zero. Alternatively, one could maximize the log likelihood directly by using a numeric optimization algorithm.

$\endgroup$
  • $\begingroup$ Thanks for reply! However, I think in the multiple-output regression situation, $y$ is actually a matrix with the dimension N X OutputDim, where N is the number of the training samples, so the term $-\frac{1}{2}y^TK_y^{-1}y$ is finally a OutputDim X OutputDim matrix instead of a scalar here, or do I misunderstand something? $\endgroup$ – mikegao88 Nov 23 '15 at 6:42
  • $\begingroup$ I think that y is a vector of scalar responses. chapter 5.4 looks at the same model as chapter 2, which involves a scalar response. the feature space is higher dimensional. $\endgroup$ – Placidia Nov 23 '15 at 17:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.