43
$\begingroup$

I understand what a Posterior is, but I'm not sure what the latter means?

How are the 2 different?

Kevin P Murphy indicated in his textbook, Machine Learning: a Probabilistic Perspective, that it is "an internal belief state". What does that really mean? I was under the impression that a Prior represents your internal belief or bias, where am I going wrong?

$\endgroup$
52
$\begingroup$

The simple difference between the two is that the posterior distribution depends on the unknown parameter $\theta$, i.e., the posterior distribution is: $$p(\theta|x)=c\times p(x|\theta)p(\theta)$$ where $c$ is the normalizing constant.

While on the other hand, the posterior predictive distribution does not depend on the unknown parameter $\theta$ because it has been integrated out, i.e., the posterior predictive distribution is: $$p(x^*|x)=\int_\Theta c\times p(x^*,\theta|x)d\theta=\int_\Theta c\times p(x^*|\theta)p(\theta|x)d\theta$$

where $x^*$ is a new unobserved random variable and is independent of $x$.

I won't dwell on the posterior distribution explanation since you say you understand it but the posterior distribution "is the distribution of an unknown quantity, treated as a random variable, conditional on the evidence obtained" (Wikipedia). So basically its the distribution that explains your unknown, random, parameter.

On the other hand, the posterior predictive distribution has a completely different meaning in that it is the distribution for future predicted data based on the data you have already seen. So the posterior predictive distribution is basically used to predict new data values.

If it helps, is an example graph of a posterior distribution and a posterior predictive distribution:

enter image description here

enter image description here

$\endgroup$
  • 3
    $\begingroup$ That posterior predictive distribution graph needs new axis labels and a caption or something. I get the idea because I know what a posterior predictive distribution is, but someone who's just figuring it out could get seriously confused. $\endgroup$ – Cyan Sep 25 '13 at 20:24
  • $\begingroup$ Thanks @BabakP could you also point me to what what distribution you used to plot the pmf of theta, and P(x*|theta) $\endgroup$ – A.D Sep 26 '13 at 3:42
  • $\begingroup$ ...cause I would like to work out the full example. $\endgroup$ – A.D Sep 26 '13 at 3:48
  • $\begingroup$ I just pretended that my posterior was a Beta(3,2). I did not actually work out anything. But of course, if you want an example, assume the likelihood is a Binomial(n,p) and the prior on p is a Beta(a,b) then you should be able to obtain that the posterior is once again a beta distribution. $\endgroup$ – user25658 Sep 26 '13 at 5:03
  • 1
    $\begingroup$ There is a wording problem. The posterior distribution does not depend on the parameter. It is a function of the parameter but one does does not need to know the true value of the parameter. The distinction is whether you want a distribution of $\theta$ or a distribution of data. $\endgroup$ – Frank Harrell Jun 24 '18 at 13:39
13
$\begingroup$

The predictive distribution is usually used when you have learned a posterior distribution for the parameter of some sort of predictive model. For example in Bayesian linear regression, you learn a posterior distribution over the w parameter of the model y=wX given some observed data X.
Then when a new unseen data point x* comes in, you want to find the distribution over possible predictions y* given the posterior distribution for w that you just learned. This distribution over possible y*'s given the posterior for w is the prediction distribution.

$\endgroup$
10
$\begingroup$

They refer to distributions of two different things.

The posterior distribution refers to the distribution of the parameter, while the predictive posterior distribution (PPD) refers to the distribution of future observations of data.

$\endgroup$
  • $\begingroup$ So, if I understand this correctly, if the true likelihood distribution (or true distribution where the data comes from) is Gaussian, then as we gather more and more data (observations), the PPD should converge towards a Gaussian (for some parameter $\theta$), while the posterior distribution should converge towards a spike at the true parameter $\theta$? $\endgroup$ – SimpleProgrammer Oct 29 '20 at 20:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.