Model evidence and Bayesian model selection

Question

I'm reading some course notes on Bayesian statistics and in one of the slides entitled 'model evidence' it writes:

$$p(y|m)=\int{p(y,\theta|m)d\theta}=\int{p(y|\theta,m)p(\theta | m)d\theta}$$ "Because we have marginalised over $\theta$ the evidence is also known as the marginal likelihood."

I have two issues here.

1. I don't understand how does $p(y,\theta|m)$ become $p(y|\theta,m)p(\theta | m)$. Is this derived from the multiplication rule of dependent events $P(A,B)=P(A|B)P(B)=P(B|A)P(A)$? If so - I don't see how. What is this linked to "we have marginalized over theta"?
2. What does $m$ really stand for? I know what it's just supposed to stand for but it's just beyond my comprehension. How is it related to model parameters?

Answer 1

1. Yes, it is. As you mentioned, the classical rule is $P(A,B) = P(A|B)P(B)$, but it can also be applied to conditional probabilities like $P(\cdot|C)$ instead of $P(\cdot)$. It then becomes

$$ P(A,B|C) = P(A|B,C)P(B|C) $$

(you just add a condition on $C$, but otherwise that's the same formula). You can then apply this formula for $A = y$, $B = \theta$, and $C = m$.

You know from the law of total probability that, if $\{B_n\}$ is a partition of the sample space, we obtain

$$ p(A) = \sum_n p(A,B_n) $$

or, using the first formula:

$$ p(A) = \sum_n p(A|B_n)p(B_n) $$

This easily extends to continuous random variables, by replacing the sum by an integral:

$$ p(A) = \int p(A|B)p(B) dB $$

The action of making $B$ "disappear" from $p(A,B)$ by integrating it over $B$ is called "marginalizing" ($B$ has been marginalized out). Once again, you can apply this formula for $A = y$, $B = \theta$, and $C = m$.

2. $m$ is the model. Your data $y$ can have been generated from a certain model $m$, and this model itself has some parameters $\theta$. In this setting, $p(y|\theta,m)$ is the probability to have data $y$ from model $m$ parametrized with $\theta$, and $p(\theta|m)$ is the prior distribution of the parameters of model $m$.

For example, imagine you are trying to fit some data using either a straight line or a parabola. Your 2 models are thus $m_2$, where data are explained as $y = a_2 x^2 + a_1 x + a_0 + \epsilon$ ($\epsilon$ is just some random noise) and its parameters are $\theta_2 = [a_2 \ a_1 \ a_0]$ ; and $m_1$, where data are explained as $y = a_1 x + a_0 + \epsilon$ and its parameters are $\theta_1 = [ a_1 \ a_0]$.

For further examples, you can have a look at this paper, where we defined different models of synapse, each with different parameters : https://www.frontiersin.org/articles/10.3389/fncom.2020.558477/full

You can also have a look at the comments here : Formal proof of Occam's razor for nested models