This issue is related to a real problem, but I've boiled down to a minimal example.

Simplest version

Suppose I get to observe two variables, $X_1$ and $X_2$. We can think of these data being generated by a mixture/hierarchical model, in which they have a 50% chance of actually being the same uniform(0, $\alpha$) and 50% chance of being two independent draws from a uniform(0, $\alpha$), where $\alpha$ is known in advance. We can more formally define this as

$Z \sim \text{Bern}(0.5)$

If $Z = 0$, $X_1 = X_2 \sim \text{Unif}(0, \alpha)$

If $Z = 1$, $X_1 \sim \text{Unif}(0, \alpha), X_2 \sim \text{Unif}(0, \alpha)$

and we observe $X_1, X_2$ but not $Z$.

Now let's suppose we observe $X_1 = X_2 = \alpha/2$. This case, we should know that they must have come from the same uniform variable (i.e., $Z = 0$), as the probability that two continuous variables are exactly equivalent is 0.

But what happens to the posterior? If we want to calculate

$P(Z = 0 | X_1 = X_2 = \alpha/2) =$ $\frac{P(Z = 0 \land X_1 = \alpha/2 \land X_2 = \alpha/2)}{ P(X_1 = \alpha/2 \land X_2 = \alpha/2)}$

Here's definitely where the issue lies, but as we normally apply Bayes Theorem, we would switch to density functions for $X_1$ and $X_2$. In that case, we would get that

$f_{X_1, X_2|Z = 0}(x_1, x_2) = \alpha^{-1}$ if $x_1 = x_2 \land x_1 \in [0, \alpha]$.


$f_{X_1, X_2|Z = 1}(x_1, x_2) = \alpha^{-2}$ if $x_1, x_2 \in [0, \alpha]$.

If we plug that back into our original problem, we get something that is clearly wrong:

$P(Z = 0 | X_1 = X_2 = \alpha/2) = \frac{0.5 \times \alpha^{-1}}{0.5 \times \alpha^{-1} + 0.5 \times \alpha^{-2}} = \frac{1}{1 + \alpha^{-1}}$

Not only is that not 1, it approaches 0 as $\alpha$ approaches 0!

Above has a clear answer and you can at least reason with it by saying that the issue is that $x_1 = x_2$ is infinitely more likely if $Z = 0$...but directly looking at the density functions does not make this obvious. In general, how should I have recognized this type of error when comparing two models of differing dimensions?


1 Answer 1


The problem presented here is not a problem of comparing models of different dimensions. The problem is that the model is incomplete. We can amend the model by defining a joint distribution for $X_1, X_2$ using a conditional distribution as follows:

$Z \sim \text{Bernoulli}(0.5)$

$X_1 \sim \text{Uniform}(0, \alpha)$

$X_2 | Z = 1, X_1=x_1 \sim \text{Uniform}(0, \alpha)$

$X_2 | Z = 0, X_1=x_1 \sim \delta_{x_1}$

$\delta_{x_1}$ is the Dirac measure or Dirac delta (density) function (point mass) at $x_1$; you can think of it as a $\text{Normal}(x_1, \sigma^2)$ distribution with $\sigma^2\to 0$. This is the key component missing from your original model.

Next we plug everything into Bayes formula

$P(Z | X_1, X_2) \propto P(Z)P(X_1 | Z)P(X_2 | Z, X_1)$

If $X_1=X_2$, then

$P(Z=1 | X_1=x_1, X_2 = x_1) \propto 0.5 (\alpha^{-1})\alpha^{-1} $

$P(Z=0 | X_1=x_1, X_2 = x_1) \propto P(Z=0, X_1=x_1, X_2 = x_1) = 0.5(\alpha^{-1}) \infty = \infty $, thus the posterior is a $\text{Bernoulli}(0.0)$.


$P(Z=1 | X_1=x_1, X_2 = x_2) \propto 0.5(\alpha^{-1})\alpha^{-1}$

$P(Z=0 | X_1=x_1, X_2 = x_2) \propto 0.5(\alpha^{-1})0 = 0$ implying the posterior is a $\text{Bernoulli}(1.0)$.

The Spike and slab (regression setting) model is an indirectly related topic that may be of interest to you. In short, it is a mixture model consisting of a discrete component and a continuous component.

  • $\begingroup$ Can you explain why the previous model is incomplete? I think the fallacy comes from the fact that we're defining a 2D density function ($f_{X_1, X_2 | Z = 0}$) that only has positive density along a single line. $\endgroup$
    – Cliff AB
    Feb 29, 2020 at 15:50
  • $\begingroup$ What you wrote is correct, and a more formal version of the statement "Above has a clear answer and you can at least reason with it by saying that the issue is that $x_1=x_2$ is infinitely more likely if $Z=0$", but I'm really interested in understanding the mistake in the naïve approach. $\endgroup$
    – Cliff AB
    Feb 29, 2020 at 15:53
  • $\begingroup$ Actually, I am not sure I am correct to state that the model is incomplete, maybe someone more knowledgeable can comment on it. With that said, your first comment points out the right idea. There is a clear dependence between $X_1$ and $X_2$, thus we should state its dependence (one or depending on the other or vice versa) mathematically, and that is where the Dirac measure came in. Once we have a condition and a marginal, we can combine them to form a joint distribution, so we no longer have a density "along a single line". In summary: we need a joint distribution. I hope that helps. $\endgroup$
    – user
    Feb 29, 2020 at 16:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.