Bayesian comparison of differing model size This issue is related to a real problem, but I've boiled down to a minimal example. 
Simplest version
Suppose I get to observe two variables, $X_1$ and $X_2$. We can think of these data being generated by a mixture/hierarchical model, in which they have a 50% chance of actually being the same uniform(0, $\alpha$) and 50% chance of being two independent draws from a uniform(0, $\alpha$), where $\alpha$ is known in advance. We can more formally define this as 
$Z \sim \text{Bern}(0.5)$
If $Z = 0$, $X_1 = X_2 \sim \text{Unif}(0, \alpha)$
If $Z = 1$, $X_1 \sim \text{Unif}(0, \alpha), X_2 \sim \text{Unif}(0, \alpha)$
and we observe $X_1, X_2$ but not $Z$. 
Now let's suppose we observe $X_1 = X_2 = \alpha/2$. This case, we should know that they must have come from the same uniform variable (i.e., $Z = 0$), as the probability that two continuous variables are exactly equivalent is 0. 
But what happens to the posterior? If we want to calculate 
$P(Z = 0 | X_1 = X_2 = \alpha/2) =$
$\frac{P(Z = 0 \land X_1 = \alpha/2 \land X_2 = \alpha/2)}{
P(X_1 = \alpha/2 \land X_2 = \alpha/2)}$
Here's definitely where the issue lies, but as we normally apply Bayes Theorem, we would switch to density functions for $X_1$ and $X_2$. In that case, we would get that 
$f_{X_1, X_2|Z = 0}(x_1, x_2) = \alpha^{-1}$ if $x_1 = x_2 \land x_1 \in [0, \alpha]$. 
Similarly, 
$f_{X_1, X_2|Z = 1}(x_1, x_2) = \alpha^{-2}$ if $x_1, x_2 \in [0, \alpha]$.
If we plug that back into our original problem, we get something that is clearly wrong:
$P(Z = 0 | X_1 = X_2 = \alpha/2) = \frac{0.5 \times \alpha^{-1}}{0.5 \times \alpha^{-1} + 0.5 \times \alpha^{-2}} = \frac{1}{1 + \alpha^{-1}}$
Not only is that not 1, it approaches 0 as $\alpha$ approaches 0!
Above has a clear answer and you can at least reason with it by saying that the issue is that $x_1 = x_2$ is infinitely more likely if $Z = 0$...but directly looking at the density functions does not make this obvious. In general, how should I have recognized this type of error when comparing two models of differing dimensions? 
 A: The problem presented here is not a problem of comparing models of different dimensions. The problem is that the model is incomplete. We can amend the model by defining a joint distribution for $X_1, X_2$ using a conditional distribution as follows: 
$Z \sim \text{Bernoulli}(0.5)$ 
$X_1 \sim \text{Uniform}(0, \alpha)$ 
$X_2 | Z = 1, X_1=x_1 \sim \text{Uniform}(0, \alpha)$
$X_2 | Z = 0, X_1=x_1 \sim \delta_{x_1}$
$\delta_{x_1}$ is the Dirac measure or Dirac delta (density) function (point mass) at $x_1$; you can think of it as a $\text{Normal}(x_1, \sigma^2)$ distribution with $\sigma^2\to 0$. This is the key component missing from your original model.
Next we plug everything into Bayes formula
$P(Z | X_1, X_2) \propto P(Z)P(X_1 | Z)P(X_2 | Z, X_1)$
If $X_1=X_2$, then 
$P(Z=1 | X_1=x_1, X_2 = x_1) \propto 0.5 (\alpha^{-1})\alpha^{-1} $
$P(Z=0 | X_1=x_1, X_2 = x_1) \propto P(Z=0, X_1=x_1, X_2 = x_1) = 0.5(\alpha^{-1}) \infty = \infty $, thus the posterior is a $\text{Bernoulli}(0.0)$.
Otherwise
$P(Z=1 | X_1=x_1, X_2 = x_2) \propto 0.5(\alpha^{-1})\alpha^{-1}$
$P(Z=0 | X_1=x_1, X_2 = x_2) \propto 0.5(\alpha^{-1})0 = 0$ implying the posterior is a $\text{Bernoulli}(1.0)$.
The Spike and slab (regression setting) model is an indirectly related topic that may be of interest to you. In short, it is a mixture model consisting of a discrete component and a continuous component.
