Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I'm reading through the Gaussian Process book and there's one section here I don't quite understand (page 11). The author says:

"the predictive distribution is given by averaging the output of all possible linear models wrt the Gaussian posterior"

$$ \begin{aligned} p(f_*|x_*,X,y) &= \int p(f_*|x_*,w)~p(w|X,y)~dw \\ &=\mathcal N\left(\frac{1}{\sigma_n^2}x^T_*A^{-1}Xy,~x^T_*A^{-1}x_*\right) \end{aligned} $$

What does this mean, exactly? I understand that the purpose of using Gaussians is to be able to calculate uncertainty for a prediction, but I'm unclear how the "averaging of the output" doesn't end up with just a mean value of the weight. And how were the parameters for the mean and covariance derived?

share|improve this question
Type "conjugate bayesian gaussian" in Google to find some informations about the derivation. – Stéphane Laurent Nov 26 '12 at 10:30

The Posterior predictive distribution is a weighted average over your hypothesis space where each hypothesis is weighted by it's posterior probability. In Bayesian analysis, beliefs are expressed as entire distributions rather than point estimates. In your example, you have a posterior distribution over all possible weights. The fully Bayesian way to make a prediction is to marginalize out the weights by integrating over your posterior. There are alternatives to this such as taking the MAP value, which is the most probable weight value under your posterior, however that is not strictly Bayesian. It might help to review a smaller, more introductory Bayesian inference problem (e.g. inferring the mean of a Gaussian with known variance) to get your head around the concept, because it is rather fundamental to the entire approach.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.