Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
I understand what a Posterior is, but I'm not sure what the latter means?
How are the 2 different?
Kevin P Murphy indicated in his textbook, Machine Learning: a Probabilistic Perspective, that it is "an internal belief state". What does that really mean? I was under the impression that a Prior represents your internal belief or bias, where am I going wrong?
The simple difference between the two is that the posterior distribution depends on the unknown parameter $\theta$, i.e., the posterior distribution is: $$p(\theta|x)=c\times p(x|\theta)p(\theta)$$ where $c$ is the normalizing constant.
While on the other hand, the posterior predictive distribution does not depend on the unknown parameter $\theta$ because it has been integrated out, i.e., the posterior predictive distribution is: $$p(x^*|x)=\int_\Theta c\times p(x^*,\theta|x)d\theta=\int_\Theta c\times p(x^*|\theta)p(\theta|x)d\theta$$
where $x^*$ is a new unobserved random variable and is independent of $x$.
I won't dwell on the posterior distribution explanation since you say you understand it but the posterior distribution "is the distribution of an unknown quantity, treated as a random variable, conditional on the evidence obtained" (Wikipedia). So basically its the distribution that explains your unknown, random, parameter.
On the other hand, the posterior predictive distribution has a completely different meaning in that it is the distribution for future predicted data based on the data you have already seen. So the posterior predictive distribution is basically used to predict new data values.
If it helps, is an example graph of a posterior distribution and a posterior predictive distribution:
The predictive distribution is usually used when you have learned a posterior distribution for the parameter of some sort of predictive model. For example in Bayesian linear regression, you learn a posterior distribution over the w parameter of the model y=wX given some observed data X.
Then when a new unseen data point x* comes in, you want to find the distribution over possible predictions y* given the posterior distribution for w that you just learned. This distribution over possible y*'s given the posterior for w is the prediction distribution.
They refer to distributions of two different things.
The posterior distribution refers to the distribution of the parameter, while the predictive posterior distribution (PPD) refers to the distribution of future observations of data.
