So classic regression methods (ridge regression, LASSO) only predict the posterior mean $E[ Y | X ]$, while Gaussian Processes give you the full posterior distribution $Y | X$.

It would be very useful if the posterior distribution had low variance in regions where the posterior mean is actually close to the true value $Y$, and high variance in other regions. But that's not what is happening (is it?). So why is it useful to estimate the posterior distribution?

  • $\begingroup$ The ridge/LASSO make the assumption that $E[Y\mid X] = \beta^T X$. Gaussian processes let $E[Y \mid X]$ be (essentially) any function. If I model $E[Y \mid X]$ using Gaussian process regression, I do not get an estimate of the distribution of $Y \mid X$ from only the GP, I get an estimate of the posterior distribution of $E[Y \mid X]$ which is a random function in the Bayesian context (i.e. it is not fixed, and so has a posterior distribution). $\endgroup$ – guy Jan 24 '14 at 21:50

The reason why the full posterior distribution is useful is because is contains all of the relevant statistical information that you can derive given the model.

Given the posterior, you can answer any questions like:

  • What is the probability that the true value for $Y$ is between $a$ and $b$?,
  • What is the expected value of some utility function $f(y)$?

given just a point estimate (the posterior mean) doesn't tell you anything about the width of the distribution (although that can often be derived), or, for non-Gaussian models, the shape of the distribution.

This also gives a precise meaning to the idea that the estimates are "accurate" in cases where the posterior distribution has low variance: you can answer the question: "what is the probability is within some range $\pm \delta$ of a point estimate?"

| cite | improve this answer | |
  • $\begingroup$ Ok so if I understand correctly, estimating the posterior distribution does not give any information regarding the true mean or the true posterior distribution. $\endgroup$ – usual me Jan 25 '14 at 7:50
  • $\begingroup$ No, it is the (inferred) distribution for the true mean. $\endgroup$ – Dave Jan 25 '14 at 14:23
  • $\begingroup$ The "full posterior distribution" is used as weight of predictive distribution in Bayesian inference, right? $\endgroup$ – avocado Jan 31 '14 at 13:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.