Predictive Distribution is an expectation?

Question

The predictive distribution is defined as follows

$ p(\hat{y} | X) = \int p(\hat{y} | X) p(\theta | X) d\theta$

where $\hat{y}$ is unobserved value, $\theta$ represents parameters and $X$ denotes observed values (i.e data set.)

Two questions:

1. Does the integral need to be computed for every new observation? or should the posterior be interpreted as distribution over all posible new observations?

2. Is the integral equivalent to an expection? i.e:

$ p(\hat{y} | X) = E_{\theta | x} [p(\hat{y} | X)]$

If so, how should this expectation be interpreted ? is it a point estimate for $\hat{y}$ or how does it somehow represent a distribution?

Answer 1

The definition of the posterior predictive distribution is,

$$ p(\hat y \mid X) = \int p(\hat y \mid \theta) p(\theta \mid X) d\theta \quad (*) $$ where $p(\hat y \mid \theta)$ is the likelihood of your model and $p(\theta \mid X)$ is the posterior distribution of $\theta$ after observing $X$.

The quantity $p(\hat y \mid X)$ can indeed be seen as an expectation.

Your model is $p(\hat y \mid \theta)$ and after observing $X$, you knowledge on the model parameter $\theta$ is represented by the posterior distribution $p(\theta \mid X)$.

From the integral above we see that $p(\hat y \mid X)$ is the expectation of $p(\hat y \mid \theta)$ given that the distribution of $\theta$ is $p(\theta \mid X)$, i.e:

$$ p(\hat y \mid X) = \mathbb E_{\theta \mid X} \left [ p(\hat y \mid \theta )\right ]. $$

Another way to see $p(\hat y \mid X)$ is think about it as the sum across $\theta$ of $p(\hat y \mid \theta )$, i.e the probability of $\hat y$ given the model is $\theta$, times your current knowledge of the probability of this model $p(\theta \mid X)$.

So it completely represents a distribution (a distribution marginalized over $\theta$).

If you wanted a point estimate of $\hat y$ you could take for example the expectation of this distribution: $$ \int \hat y p(\hat y \mid X)d\hat y. $$

Last, if I understand your first question, in the general case if you want to evaluate $p(\hat y \mid X)$ for different values of $\hat y $ you will need to recompute the integral in $(*)$ each time.