6
$\begingroup$

In the Bayesian framework, to me, it seems that the marginal likelihood and the prior predictive distribution/probability are equal. Is that the case? Or maybe this just holds for single data points? Why differ between these two terms?

Marginal likelihood (evidence): $$ p(\mathbb{X}|\alpha) = \int_\theta p(\mathbb{X}|\theta) \, p(\theta|\alpha)\ \operatorname{d}\!\theta $$

Prior predictive distribution: $$ p(\tilde{x}|\alpha) = \int_{\theta} p(\tilde{x}|\theta) \, p(\theta|\alpha) \operatorname{d}\!\theta $$

$\endgroup$
  • $\begingroup$ For example, when there are two parameters, one being the nuisance parameter and the other being the parameter of interest, I use the term marginal likelihood for the likelihood integrated over the conditional prior distribution of the nuisance parameter given the parameter of interest. What have you seen exactly ? The likelihood is about a parameter and is not a probability distribution. $\endgroup$ – Stéphane Laurent Apr 21 '15 at 13:19
  • $\begingroup$ You are talking about frequentist ways of seeing the marginal likelihood not the Bayesian way (i.e, evidence). I have added the definitions to the question. $\endgroup$ – fsociety Apr 21 '15 at 13:23
  • $\begingroup$ In view of your formulas, you have two parameters $\alpha$ and $\theta$ ? If so, this is exactly what I said in my first comment. $\endgroup$ – Stéphane Laurent Apr 21 '15 at 14:14
  • $\begingroup$ Yes, you marginalize $\theta$ out. I still do not have an answer to the question though. $\endgroup$ – fsociety Apr 21 '15 at 14:17
  • $\begingroup$ You are right, my answer has nothing to do with the "marginal likelihood" in the sense of your question. I prefer not to delete it, because it could be useful for someone who wonders about this terminology. $\endgroup$ – Stéphane Laurent Jul 21 '15 at 18:18
3
$\begingroup$

I'm assuming $\alpha$ contains the values that define your prior for $\theta$. When this is the case, we typically omit $\alpha$ from the notation and have the marginal likelihood $$p(\mathbb{X}) = \int p(\mathbb{X}|\theta) p(\theta) d\theta.$$ The prior predictive distribution is not well defined in that you haven't told me what it is that you want predicted, e.g. the prior predictive distribution is different when predicting a single data point and predicting a set of observations. In the notation, this is confusing because $p(\tilde{x}|\theta)$ is different depending on what $\tilde{x}$ is.

If you want to predict data that has exactly the same structure as the data you observed, then the marginal likelihood is just the prior predictive distribution for data of this structure evaluated at the data you observed, i.e. the marginal likelihood is a number whereas the prior predictive distribution has a probability density (or mass) function.

$\endgroup$
  • $\begingroup$ Thanks, that makes sense. So let us assume that in both cases I look at the same set of observations. Then, they seem to resemble a similar thing with the difference you pointed out. I would end up with a single value for the marginal likelihood and a probability distribution over the single data values for the prior predictive distribution, right? So, by and large, the notation is slightly irritating. $\endgroup$ – fsociety Apr 21 '15 at 16:31
3
$\begingroup$

For a parametric model ${\cal M} = \{p(\cdot \mid \theta, \alpha)\}$ with two parameters $\theta$ and $\alpha$ equipped with a prior distribution $\pi(\theta, \alpha)$ then the ("joint") likelihood on $(\theta, \alpha)$ after $x$ has been observed is defined by $$L(\theta, \alpha \mid x) \overset{\theta,\alpha}{\propto} p(x \mid \theta, \alpha).$$ See here about my notation $\overset{\theta,\alpha}{\propto}$.

The marginal likelihood on $\alpha$ is obtained by integrating the joint likelihood over the conditional prior distribution $\pi(\theta \mid \alpha)$: $$\tilde L(\alpha \mid x) \overset{\alpha}{\propto} \int L(\theta,\alpha \mid x) \pi(\theta \mid \alpha) d\theta.$$ This is nothing but the "ordinary" likelihood for a new model $\tilde{\cal M} = \{\tilde p(\cdot \mid \alpha)\}$ with parameter $\alpha$, obtained by integrating the original sampling distribution over the conditional prior distribution $\pi(\theta \mid \alpha)$: $$\tilde p(x \mid \alpha) = \int p(x \mid \theta, \alpha)\pi(\theta \mid \alpha) d\theta$$ which is also the conditional prior predictive distribution (of $x$ given $\alpha$). Using the marginal prior distribution $\pi(\alpha)$ of $\alpha$ for this model yields exactly the same posterior distribution: $$\pi(\alpha \mid x) \overset{\alpha}{\propto} \pi(\alpha)\tilde L(\alpha \mid x).$$

To sum up, the marginal likelihood is the likelihood of the model whose sampling distribution is the conditional prior predictive distribution.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.