How to compute prediction error from Relevance Vector Machine and Gaussian Process Regression?  Does anyone know how to construct a confidence interval for predicting a new test value given a trained Relevance Vector Machine (rvm) and/or Gaussian Process Regression (gausspr) using the kernlab R package?
More specifically, how do I get:


*

*The standard error/deviation (variance) of a new test point;

*The parameters estimates of posterior distribution of the parameters?
I would appreciate if anyone could point to a document that discuss how to obtain/calculate the above, from the output of function call (rvm or gausspr).
 A: Ok, here's the scoop: first off, have a look at my class in Berkeley. The slides for GP regression should be up shortly at http://alex.smola.org/teaching/berkeley2012 where I discuss this in more detail.
For a GP regression you can integrate out the latent GP and since the observation model and the underlying GP are both Gaussian, you get a Gaussian again. In math this means that
$$y = t + \epsilon.$$
Here $t$ is drawn from a GP and $\epsilon$ are independent Gaussian random variables. Once you have the GP representation for $y$, you can simply condition on the observed terms to get the residual variance for the rest. This yields something of the form
$$E[y'|y] = \mu' + K_{x'x} K_{xx}^{-1} (y - \mu)$$
and the variance is 
$$K_{x'x'} - K_{x'x} K_{xx}^{-1} K_{xx'}.$$
Obviously here we added the noise for $\epsilon$ to the main diagonal. For the relevance vector machine this isn't quite so trivial since it assumes a rather different noise model. So you cannot integrate out the latent variables explicitly. Have a look at my book 'Learning with Kernels' or alternatively Mike Tipping's original paper for the details.
