How to compute prediction error from Relevance Vector Machine and Gaussian Process Regression?

Does anyone know how to construct a confidence interval for predicting a new test value given a trained Relevance Vector Machine (rvm) and/or Gaussian Process Regression (gausspr) using the kernlab R package?

More specifically, how do I get:

1. The standard error/deviation (variance) of a new test point;

2. The parameters estimates of posterior distribution of the parameters?

I would appreciate if anyone could point to a document that discuss how to obtain/calculate the above, from the output of function call (rvm or gausspr).

1 Answer

Ok, here's the scoop: first off, have a look at my class in Berkeley. The slides for GP regression should be up shortly at http://alex.smola.org/teaching/berkeley2012 where I discuss this in more detail.

For a GP regression you can integrate out the latent GP and since the observation model and the underlying GP are both Gaussian, you get a Gaussian again. In math this means that

$$y = t + \epsilon.$$

Here $t$ is drawn from a GP and $\epsilon$ are independent Gaussian random variables. Once you have the GP representation for $y$, you can simply condition on the observed terms to get the residual variance for the rest. This yields something of the form

$$E[y'|y] = \mu' + K_{x'x} K_{xx}^{-1} (y - \mu)$$

and the variance is

$$K_{x'x'} - K_{x'x} K_{xx}^{-1} K_{xx'}.$$

Obviously here we added the noise for $\epsilon$ to the main diagonal. For the relevance vector machine this isn't quite so trivial since it assumes a rather different noise model. So you cannot integrate out the latent variables explicitly. Have a look at my book 'Learning with Kernels' or alternatively Mike Tipping's original paper for the details.