# Bayesian comparison of differing model size

This issue is related to a real problem, but I've boiled down to a minimal example.

Simplest version

Suppose I get to observe two variables, $$X_1$$ and $$X_2$$. We can think of these data being generated by a mixture/hierarchical model, in which they have a 50% chance of actually being the same uniform(0, $$\alpha$$) and 50% chance of being two independent draws from a uniform(0, $$\alpha$$), where $$\alpha$$ is known in advance. We can more formally define this as

$$Z \sim \text{Bern}(0.5)$$

If $$Z = 0$$, $$X_1 = X_2 \sim \text{Unif}(0, \alpha)$$

If $$Z = 1$$, $$X_1 \sim \text{Unif}(0, \alpha), X_2 \sim \text{Unif}(0, \alpha)$$

and we observe $$X_1, X_2$$ but not $$Z$$.

Now let's suppose we observe $$X_1 = X_2 = \alpha/2$$. This case, we should know that they must have come from the same uniform variable (i.e., $$Z = 0$$), as the probability that two continuous variables are exactly equivalent is 0.

But what happens to the posterior? If we want to calculate

$$P(Z = 0 | X_1 = X_2 = \alpha/2) =$$ $$\frac{P(Z = 0 \land X_1 = \alpha/2 \land X_2 = \alpha/2)}{ P(X_1 = \alpha/2 \land X_2 = \alpha/2)}$$

Here's definitely where the issue lies, but as we normally apply Bayes Theorem, we would switch to density functions for $$X_1$$ and $$X_2$$. In that case, we would get that

$$f_{X_1, X_2|Z = 0}(x_1, x_2) = \alpha^{-1}$$ if $$x_1 = x_2 \land x_1 \in [0, \alpha]$$.

Similarly,

$$f_{X_1, X_2|Z = 1}(x_1, x_2) = \alpha^{-2}$$ if $$x_1, x_2 \in [0, \alpha]$$.

If we plug that back into our original problem, we get something that is clearly wrong:

$$P(Z = 0 | X_1 = X_2 = \alpha/2) = \frac{0.5 \times \alpha^{-1}}{0.5 \times \alpha^{-1} + 0.5 \times \alpha^{-2}} = \frac{1}{1 + \alpha^{-1}}$$

Not only is that not 1, it approaches 0 as $$\alpha$$ approaches 0!

Above has a clear answer and you can at least reason with it by saying that the issue is that $$x_1 = x_2$$ is infinitely more likely if $$Z = 0$$...but directly looking at the density functions does not make this obvious. In general, how should I have recognized this type of error when comparing two models of differing dimensions?

The problem presented here is not a problem of comparing models of different dimensions. The problem is that the model is incomplete. We can amend the model by defining a joint distribution for $$X_1, X_2$$ using a conditional distribution as follows:

$$Z \sim \text{Bernoulli}(0.5)$$

$$X_1 \sim \text{Uniform}(0, \alpha)$$

$$X_2 | Z = 1, X_1=x_1 \sim \text{Uniform}(0, \alpha)$$

$$X_2 | Z = 0, X_1=x_1 \sim \delta_{x_1}$$

$$\delta_{x_1}$$ is the Dirac measure or Dirac delta (density) function (point mass) at $$x_1$$; you can think of it as a $$\text{Normal}(x_1, \sigma^2)$$ distribution with $$\sigma^2\to 0$$. This is the key component missing from your original model.

Next we plug everything into Bayes formula

$$P(Z | X_1, X_2) \propto P(Z)P(X_1 | Z)P(X_2 | Z, X_1)$$

If $$X_1=X_2$$, then

$$P(Z=1 | X_1=x_1, X_2 = x_1) \propto 0.5 (\alpha^{-1})\alpha^{-1}$$

$$P(Z=0 | X_1=x_1, X_2 = x_1) \propto P(Z=0, X_1=x_1, X_2 = x_1) = 0.5(\alpha^{-1}) \infty = \infty$$, thus the posterior is a $$\text{Bernoulli}(0.0)$$.

Otherwise

$$P(Z=1 | X_1=x_1, X_2 = x_2) \propto 0.5(\alpha^{-1})\alpha^{-1}$$

$$P(Z=0 | X_1=x_1, X_2 = x_2) \propto 0.5(\alpha^{-1})0 = 0$$ implying the posterior is a $$\text{Bernoulli}(1.0)$$.

The Spike and slab (regression setting) model is an indirectly related topic that may be of interest to you. In short, it is a mixture model consisting of a discrete component and a continuous component.

• Can you explain why the previous model is incomplete? I think the fallacy comes from the fact that we're defining a 2D density function ($f_{X_1, X_2 | Z = 0}$) that only has positive density along a single line. Feb 29, 2020 at 15:50
• What you wrote is correct, and a more formal version of the statement "Above has a clear answer and you can at least reason with it by saying that the issue is that $x_1=x_2$ is infinitely more likely if $Z=0$", but I'm really interested in understanding the mistake in the naïve approach. Feb 29, 2020 at 15:53
• Actually, I am not sure I am correct to state that the model is incomplete, maybe someone more knowledgeable can comment on it. With that said, your first comment points out the right idea. There is a clear dependence between $X_1$ and $X_2$, thus we should state its dependence (one or depending on the other or vice versa) mathematically, and that is where the Dirac measure came in. Once we have a condition and a marginal, we can combine them to form a joint distribution, so we no longer have a density "along a single line". In summary: we need a joint distribution. I hope that helps.
– user
Feb 29, 2020 at 16:10