What is the role of model likelihood?

Question

$$ p(\theta \mid X, M)=\frac{p(X \mid \theta, M) p(\theta \mid M)}{p(X \mid M)} $$

Just proved this is upper probability equation is correct. It differs a bit from the ordinary Bayesian rule:

$$ p(\theta \mid X)=\frac{p(X \mid \theta) p(\theta)}{p(X)} $$

since it introduces model likelihood: $p(X \mid M)$. How you would interpret model likelihood in comparison to $p(X \mid \theta, M)$ which should be data likelihood, but also parametrized by model and parameters.

Searched almost anything on this site under "model likelihood" and few links didn't helped much.

The problem: I associated word likelihood with the word data, but now there is model likelihood that is not about parameters. I know model is structure and parameters, so I am assuming model likelihood has to have something with the structure of the model.

Also the part of my confusion is how we approach to evidence since model likelihood is officially the evidence, and I know the evidence is either observations or marginal likelihood of observations.

Help me define and reason on model likelihood term, why may be needed and in what context.

Answer 1

TL;DR: what you found, the marginal likelihood, is useful for model comparison. It's the total evidence for this class of model, over all its possible parameter settings, given the data.

Just proved this is upper probability equation is correct. It differs a bit from the ordinary Bayesian rule:

In fact, your first equation is a bit better! Compared to the commonplace one, it makes assumptions about the model explicit.

it introduces model likelihood: 𝑝(𝑋∣𝑀)

Let's call this the marginal likelihood today. ('Evidence' is also a fine name for it.)

How you would interpret model likelihood in comparison to 𝑝(𝑋∣𝜃,𝑀) which should be data likelihood, but also parametrized by model and parameters. How you would interpret model likelihood in comparison to 𝑝(𝑋∣𝜃,𝑀) which should be data likelihood, but also parametrized by model and parameters.

Well, $p(X \mid \theta, M)$ is (terminologically) the likelihood of the parameters, not of the data. (If you want to mention the associated data, you may say "the likelihood of the parameters given the data".) In other words, when we talk about likelihood, $X$ is fixed, and we're varying $\theta$.

By contrast, $p(X \mid M)$ which we call the marginal likelihood, is the total evidence for this model, across all possible settings of its parameters $\theta$. I chose to call it the marginal likelihood to remind you that it is $\int_\theta p(X \mid \theta, M) p(\theta \mid M)~ \mathrm{d}\theta$. The big point is: how well can this class of model explain the data?

The marginal likelihood is necessary to normalize your equation, but when we optimize it's common to ignore it. (After all, it's a constant. So we can just optimize $\mathrm{likelihood} \times \mathrm{prior}$.) It starts to matter when we do model comparison, where the parameter space may be different.

Model comparison: an example.

The marginal likelihood is useful for model comparison. Imagine a simple coin-flipping problem, where model $M_0$ is that it's biased with parameter $p_0=0.3$ and model $M_1$ is that it's biased with an unknown parameter $p_1$. For $M_0$, we only integrate over the single possible value. For $M_1$, we integrate over all possible values from $0$ to $1$.

$$ \begin{align} p(M_1 \mid X) &= \frac{P(X \mid M_1) P(M_1)}{P(X)} \\ P(X) &= P(X \mid M_1) P(M_1) + P(X \mid M_0) P(M_0) \\ \end{align} $$

With these equations, we can see how much the data favor $M_1$, which is the evidence for model $M_1$.