Confused by a simple setting in Bayesian inference

Question

I want to use Bayesian approach to test whether a single data point $x$ came from model $M_1$ or model $M_2$.

I am having difficult time to get my head around this very basic setting. I make a few steps a long the way and then I get stuck \ confused.

---

So the two models are:

$$ M_1: X \sim N(0, 1)\,, $$ $$ M_2: X \sim N(\mu, 1)~~~ \text{ with }~~ \mu \sim U[1, 2]\,. $$

Where $N(\mu, \sigma^2)$ stands for Normal distribution with mean $\mu$ and variance $\sigma^2$, and $U[a, b]$ is uniform distribution.

---

My attempt to find the posterior odds.

By Bayesian formula

$$ \frac{P(M_1|x)}{P(M_2|x)} = \frac{P(x|M_1)}{P(x|M_2)}\frac{P(M_1)}{P(M_2)}\,. $$

At this point I need to introduce priors for $M_1$ and $M_2$ let those be $\pi_1$ and $\pi_2 = (1- \pi_1)$, this leads to

$$ \frac{P(M_1|x)}{P(M_2|x)} = \frac{P(x|M_1)}{P(x|M_2)}\frac{\pi_1}{ (1- \pi_1)}\,. $$

Here I already get confused - formally probability of $P(X = x|M_1) = 0$ , any way, I do continue

$$ P(X = x|M_2) = \int f_{(X|\mu)}(x)f_\mu(\mu)d\mu \,, $$

where $f_{(X|\mu)}(\cdot)$ is conditional probability density of $X$ given $\mu$ and $f_\mu(\cdot)$ is probability density of $\mu$.

$f_\mu(\cdot)$ is uniform on $[1,2]$. Thus,

$$ P(X = x|M_2) = \int f_{(X|\mu)}(x)f_\mu(\mu)d\mu = \int_1^2 \frac{1}{\sqrt{2\pi}}e^{-\frac{(x-\mu)^2}{2}}d\mu = C \neq 0\,. $$

Overall

$$ \frac{P(M_1|x)}{P(M_2|x)} = \frac{P(x|M_1)}{P(x|M_2)}\frac{\pi_1}{ (1- \pi_1)} = \frac{0}{C}\frac{\pi_1}{ (1- \pi_1)} \equiv 0\,. \quad(\textbf{?}) $$

So, regardless of $x$ the odds in favor of $M_0$ are zero.

---

Am I having not enough coffee? did I get the Bayesian development above all wrong, should I have used likelihood function instead of probability (that is to look at densities not probabilities)?

I would appreciate any help, thanks!

Answer 1

For continuous models, the Bayes factor is defined as the ratio of marginal likelihoods (marginal density functions of the data):

$$P(M_i\vert x) = \int p(x\vert\theta_i, M_i)\pi(\theta_i)d\theta_i \neq 0,$$

where $p$ denotes the likelihood function (joint density of the data given the parameters of model $M_i$).

See:

Kass, Robert E., and Adrian E. Raftery. "Bayes factors." Journal of the american statistical association 90.430 (1995): 773-795.

EDIT:

Regarding the derivation of the formula. Recall that discrete and continuous variables cannot be treated the same way. Thus, the conditional probability is given by

$$P(M_i\vert x) = \dfrac{P(M_i)f(x\vert M_i)}{f(x)}.$$

Using the Law of total probability:

$$f(x\vert M_i) = \int f(x\vert M_i,\theta_i)\pi(\theta_i)d\theta_i.$$ Thus:

$$P(M_i\vert x) = \dfrac{P(M_i)\int f(x\vert M_i,\theta_i)\pi(\theta_i)d\theta_i}{f(x)}.$$

Finally:

$$\frac{P(M_1\vert x)}{P(M_2\vert x)} = \dfrac{P(M_1)\int f(x\vert M_1,\theta_1)\pi(\theta_1)d\theta_1}{P(M_2)\int f(x\vert M_2,\theta_2)\pi(\theta_2)d\theta_2}.$$

$\theta_i$ represent the parameters associated to model $M_i$.

Answer 2

In model $M_1$, there is no parameter, hence no need for a prior distribution. The marginal density of $X$ under model $M_1$ is thus the normal $N(0,1)$ in this case. The marginal density under model $M_2$ is indeed the integral $$ \int f_{X}(x|\mu)f_\mu(\mu)\text{d}\mu = \int_1^2 \frac{1}{\sqrt{2\pi}}e^{-\frac{(x-\mu)^2}{2}}\text{d}\mu=\Phi(2-x)-\Phi(1-x)$$

The Bayes factor is an odds ratio associated with the posterior probability over the model index (1 versus 2), when the parameter (if any) is integrated out. Hence it stems from Bayes' theorem.

Note: You should not use the notation $P(X=x|M)$ for continuous random variables, as this probability is always zero.