1
$\begingroup$

$$ p(\theta \mid X, M)=\frac{p(X \mid \theta, M) p(\theta \mid M)}{p(X \mid M)} $$

Just proved this is upper probability equation is correct. It differs a bit from the ordinary Bayesian rule:

$$ p(\theta \mid X)=\frac{p(X \mid \theta) p(\theta)}{p(X)} $$

since it introduces model likelihood: $p(X \mid M)$. How you would interpret model likelihood in comparison to $p(X \mid \theta, M)$ which should be data likelihood, but also parametrized by model and parameters.

Searched almost anything on this site under "model likelihood" and few links didn't helped much.

The problem: I associated word likelihood with the word data, but now there is model likelihood that is not about parameters. I know model is structure and parameters, so I am assuming model likelihood has to have something with the structure of the model.

Also the part of my confusion is how we approach to evidence since model likelihood is officially the evidence, and I know the evidence is either observations or marginal likelihood of observations.

Help me define and reason on model likelihood term, why may be needed and in what context.

$\endgroup$
3
  • 2
    $\begingroup$ Presumably $p(X\mid M)=\int_\theta p(X \mid \theta, M) \,p(\theta \mid M)\, d\theta$ $\endgroup$
    – Henry
    Mar 25 at 18:15
  • 1
    $\begingroup$ Your first equation looks like your second, except that the first is explicitly with the assumption that model $M$ applies $\endgroup$
    – Henry
    Mar 25 at 18:17
  • $\begingroup$ Sure somehow parameters are marginalized out in your first equation @Henry, even though intuitively they should be part of the model. I also searched the internet, and model likelihood is hiding from me. I have a good spirit that it is not so hard question still maybe is less used. $\endgroup$
    – Good Luck
    Mar 25 at 18:25
6
$\begingroup$

TL;DR: what you found, the marginal likelihood, is useful for model comparison. It's the total evidence for this class of model, over all its possible parameter settings, given the data.

Just proved this is upper probability equation is correct. It differs a bit from the ordinary Bayesian rule:

In fact, your first equation is a bit better! Compared to the commonplace one, it makes assumptions about the model explicit.

it introduces model likelihood: 𝑝(𝑋∣𝑀)

Let's call this the marginal likelihood today. ('Evidence' is also a fine name for it.)

How you would interpret model likelihood in comparison to 𝑝(𝑋∣𝜃,𝑀) which should be data likelihood, but also parametrized by model and parameters. How you would interpret model likelihood in comparison to 𝑝(𝑋∣𝜃,𝑀) which should be data likelihood, but also parametrized by model and parameters.

Well, $p(X \mid \theta, M)$ is (terminologically) the likelihood of the parameters, not of the data. (If you want to mention the associated data, you may say "the likelihood of the parameters given the data".) In other words, when we talk about likelihood, $X$ is fixed, and we're varying $\theta$.

By contrast, $p(X \mid M)$ which we call the marginal likelihood, is the total evidence for this model, across all possible settings of its parameters $\theta$. I chose to call it the marginal likelihood to remind you that it is $\int_\theta p(X \mid \theta, M) p(\theta \mid M)~ \mathrm{d}\theta$. The big point is: how well can this class of model explain the data?

The marginal likelihood is necessary to normalize your equation, but when we optimize it's common to ignore it. (After all, it's a constant. So we can just optimize $\mathrm{likelihood} \times \mathrm{prior}$.) It starts to matter when we do model comparison, where the parameter space may be different.

Model comparison: an example.

The marginal likelihood is useful for model comparison. Imagine a simple coin-flipping problem, where model $M_0$ is that it's biased with parameter $p_0=0.3$ and model $M_1$ is that it's biased with an unknown parameter $p_1$. For $M_0$, we only integrate over the single possible value. For $M_1$, we integrate over all possible values from $0$ to $1$.

$$ \begin{align} p(M_1 \mid X) &= \frac{P(X \mid M_1) P(M_1)}{P(X)} \\ P(X) &= P(X \mid M_1) P(M_1) + P(X \mid M_0) P(M_0) \\ \end{align} $$

With these equations, we can see how much the data favor $M_1$, which is the evidence for model $M_1$.

$\endgroup$
8
  • $\begingroup$ so what is the role of model likelihood? $\endgroup$
    – Good Luck
    Mar 26 at 10:42
  • $\begingroup$ With $\int_\theta p(X \mid \theta, M) ~ \mathrm{d}\theta$ parameters are marginalized. Are parameters part of the model still? $\endgroup$
    – Good Luck
    Mar 26 at 10:43
  • $\begingroup$ Of course parameters are part of the model. The marginal likelihood considers the evidence for this model, across all possible settings of its parameters. $\endgroup$ Mar 26 at 11:56
  • $\begingroup$ “What is the role?” Normalization and model comparison. $\endgroup$ Mar 26 at 11:57
  • 1
    $\begingroup$ This is the very first time you’ve written that expression. $p(M \mid \theta)$ doesn’t make sense semantically because the set of parameters is determined by the choice of model. Go back to the coin flip example. $M_0$ has no parameters. $M_1$ has 1. I couldn’t reasonably ask for $p(M_0 \mid \theta_1)$, because $\theta_1$ is a parameter of model $M_1$. $\endgroup$ Mar 26 at 12:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.