Marginal likelihood vs. prior predictive probability

Question

In the Bayesian framework, to me, it seems that the marginal likelihood and the prior predictive distribution/probability are equal. Is that the case? Or maybe this just holds for single data points? Why differ between these two terms?

Marginal likelihood (evidence): $$ p(\mathbb{X}|\alpha) = \int_\theta p(\mathbb{X}|\theta) \, p(\theta|\alpha)\ \operatorname{d}\!\theta $$

Prior predictive distribution: $$ p(\tilde{x}|\alpha) = \int_{\theta} p(\tilde{x}|\theta) \, p(\theta|\alpha) \operatorname{d}\!\theta $$

Answer 1

I'm assuming $\alpha$ contains the values that define your prior for $\theta$. When this is the case, we typically omit $\alpha$ from the notation and have the marginal likelihood $$p(\mathbb{X}) = \int p(\mathbb{X}|\theta) p(\theta) d\theta.$$ The prior predictive distribution is not well defined in that you haven't told me what it is that you want predicted, e.g. the prior predictive distribution is different when predicting a single data point and predicting a set of observations. In the notation, this is confusing because $p(\tilde{x}|\theta)$ is different depending on what $\tilde{x}$ is.

If you want to predict data that has exactly the same structure as the data you observed, then the marginal likelihood is just the prior predictive distribution for data of this structure evaluated at the data you observed, i.e. the marginal likelihood is a number whereas the prior predictive distribution has a probability density (or mass) function.

Answer 2

For a parametric model ${\cal M} = \{p(\cdot \mid \theta, \alpha)\}$ with two parameters $\theta$ and $\alpha$ equipped with a prior distribution $\pi(\theta, \alpha)$ then the ("joint") likelihood on $(\theta, \alpha)$ after $x$ has been observed is defined by $$L(\theta, \alpha \mid x) \overset{\theta,\alpha}{\propto} p(x \mid \theta, \alpha).$$ See here about my notation $\overset{\theta,\alpha}{\propto}$.

The marginal likelihood on $\alpha$ is obtained by integrating the joint likelihood over the conditional prior distribution $\pi(\theta \mid \alpha)$: $$\tilde L(\alpha \mid x) \overset{\alpha}{\propto} \int L(\theta,\alpha \mid x) \pi(\theta \mid \alpha) d\theta.$$ This is nothing but the "ordinary" likelihood for a new model $\tilde{\cal M} = \{\tilde p(\cdot \mid \alpha)\}$ with parameter $\alpha$, obtained by integrating the original sampling distribution over the conditional prior distribution $\pi(\theta \mid \alpha)$: $$\tilde p(x \mid \alpha) = \int p(x \mid \theta, \alpha)\pi(\theta \mid \alpha) d\theta$$ which is also the conditional prior predictive distribution (of $x$ given $\alpha$). Using the marginal prior distribution $\pi(\alpha)$ of $\alpha$ for this model yields exactly the same posterior distribution: $$\pi(\alpha \mid x) \overset{\alpha}{\propto} \pi(\alpha)\tilde L(\alpha \mid x).$$

To sum up, the marginal likelihood is the likelihood of the model whose sampling distribution is the conditional prior predictive distribution.