5
$\begingroup$

I want to use Bayesian approach to test whether a single data point $x$ came from model $M_1$ or model $M_2$.

I am having difficult time to get my head around this very basic setting. I make a few steps a long the way and then I get stuck \ confused.


So the two models are:

$$ M_1: X \sim N(0, 1)\,, $$ $$ M_2: X \sim N(\mu, 1)~~~ \text{ with }~~ \mu \sim U[1, 2]\,. $$

Where $N(\mu, \sigma^2)$ stands for Normal distribution with mean $\mu$ and variance $\sigma^2$, and $U[a, b]$ is uniform distribution.


My attempt to find the posterior odds.

By Bayesian formula

$$ \frac{P(M_1|x)}{P(M_2|x)} = \frac{P(x|M_1)}{P(x|M_2)}\frac{P(M_1)}{P(M_2)}\,. $$

At this point I need to introduce priors for $M_1$ and $M_2$ let those be $\pi_1$ and $\pi_2 = (1- \pi_1)$, this leads to

$$ \frac{P(M_1|x)}{P(M_2|x)} = \frac{P(x|M_1)}{P(x|M_2)}\frac{\pi_1}{ (1- \pi_1)}\,. $$

Here I already get confused - formally probability of $P(X = x|M_1) = 0$ , any way, I do continue

$$ P(X = x|M_2) = \int f_{(X|\mu)}(x)f_\mu(\mu)d\mu \,, $$

where $f_{(X|\mu)}(\cdot)$ is conditional probability density of $X$ given $\mu$ and $f_\mu(\cdot)$ is probability density of $\mu$.

$f_\mu(\cdot)$ is uniform on $[1,2]$. Thus,

$$ P(X = x|M_2) = \int f_{(X|\mu)}(x)f_\mu(\mu)d\mu = \int_1^2 \frac{1}{\sqrt{2\pi}}e^{-\frac{(x-\mu)^2}{2}}d\mu = C \neq 0\,. $$

Overall

$$ \frac{P(M_1|x)}{P(M_2|x)} = \frac{P(x|M_1)}{P(x|M_2)}\frac{\pi_1}{ (1- \pi_1)} = \frac{0}{C}\frac{\pi_1}{ (1- \pi_1)} \equiv 0\,. \quad(\textbf{?}) $$

So, regardless of $x$ the odds in favor of $M_0$ are zero.


Am I having not enough coffee? did I get the Bayesian development above all wrong, should I have used likelihood function instead of probability (that is to look at densities not probabilities)?

I would appreciate any help, thanks!

$\endgroup$
5
$\begingroup$

For continuous models, the Bayes factor is defined as the ratio of marginal likelihoods (marginal density functions of the data):

$$P(M_i\vert x) = \int p(x\vert\theta_i, M_i)\pi(\theta_i)d\theta_i \neq 0,$$

where $p$ denotes the likelihood function (joint density of the data given the parameters of model $M_i$).

See:

Kass, Robert E., and Adrian E. Raftery. "Bayes factors." Journal of the american statistical association 90.430 (1995): 773-795.

EDIT:

Regarding the derivation of the formula. Recall that discrete and continuous variables cannot be treated the same way. Thus, the conditional probability is given by

$$P(M_i\vert x) = \dfrac{P(M_i)f(x\vert M_i)}{f(x)}.$$

Using the Law of total probability:

$$f(x\vert M_i) = \int f(x\vert M_i,\theta_i)\pi(\theta_i)d\theta_i.$$ Thus:

$$P(M_i\vert x) = \dfrac{P(M_i)\int f(x\vert M_i,\theta_i)\pi(\theta_i)d\theta_i}{f(x)}.$$

Finally:

$$\frac{P(M_1\vert x)}{P(M_2\vert x)} = \dfrac{P(M_1)\int f(x\vert M_1,\theta_1)\pi(\theta_1)d\theta_1}{P(M_2)\int f(x\vert M_2,\theta_2)\pi(\theta_2)d\theta_2}.$$

$\theta_i$ represent the parameters associated to model $M_i$.

$\endgroup$
  • $\begingroup$ Thnks! but I still not quite understand, there is no explanation in the referenced paper, and I could not find anywhere else... When you say that "Bayes factor is defined to be" so and so... How come it is definition dependent? Isn't it something that stems from Bayes theorem? I am trying to see how can I get the probability $P(M_i |x)$ to equal the integral on the RHS, so $P(M_i | x) = \frac{P(M_i, x)}{P(x)} = \frac{P(M_i,\theta_i x)}{P(x)} = \frac{P(x|M_i, \theta_i) P(M_i, \theta_i)}{P(x)}$ but I don't readily see how I get the RHS, and what is the exact link between $\theta_i$ and $M_i$. $\endgroup$ – them Aug 23 '16 at 10:41
  • $\begingroup$ @user90793 See my edit. $\endgroup$ – Logia Aug 23 '16 at 11:12
  • $\begingroup$ Another way is to imagine your observation being "the value is between $x$ and $x+\Delta$" instead of "the value is $x$" and then taking the limit $\Delta\to 0$. Then you don't have to use point probabilities of continuous functions, and the probabilities are not $0$. In the end, this gives the same result. (Because in the limit, the probabilities are $\int_x^{x+\Delta} p(\xi) d\xi \approx p(x)\Delta$ and the $\Delta$s factor out.) $\endgroup$ – JiK Aug 23 '16 at 13:14
  • $\begingroup$ @Logia: you forgot a $P(M_i)$ in the first expression of $P(M_i|x)$... $\endgroup$ – Xi'an Aug 24 '16 at 0:53
1
$\begingroup$

In model $M_1$, there is no parameter, hence no need for a prior distribution. The marginal density of $X$ under model $M_1$ is thus the normal $N(0,1)$ in this case. The marginal density under model $M_2$ is indeed the integral $$ \int f_{X}(x|\mu)f_\mu(\mu)\text{d}\mu = \int_1^2 \frac{1}{\sqrt{2\pi}}e^{-\frac{(x-\mu)^2}{2}}\text{d}\mu=\Phi(2-x)-\Phi(1-x)$$

The Bayes factor is an odds ratio associated with the posterior probability over the model index (1 versus 2), when the parameter (if any) is integrated out. Hence it stems from Bayes' theorem.

Note: You should not use the notation $P(X=x|M)$ for continuous random variables, as this probability is always zero.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.