Simple alternative to reversible jump for nested models? Suppose that we have a model such that $p(y\mid k, \theta_1,\dots,\theta_{k_\text{max}})$ depends only on $k,\theta_1,\dots,\theta_k$. Hence, as $k$ assumes the values $1,\dots,k_\text{max}$, we have a family of nested models. With suitable priors for $k$ and the $\theta_j$'s, instead of using Green's reversible jump scheme, I was thinking if an iteration of the following much simpler procedure could work:

*

*Sample $k$ from the (available) full conditional distribution $k\mid y,\theta_1,\dots,\theta_{k_\text{max}}$.

*Do a Metropolis-Hastings step, sampling en bloc only $\theta_1,\dots,\theta_k$.

*Repeat.

Is this already used in the literature? If not, what is the catch? Does it sample the space of nested models incorrectly or inefficiently?
 A: This paper Bayesian analysis of mixture models with an unknown number of components—an alternative to reversible jump methods by Matthew Stephens is somewhat along those lines

However, our approach appears the more natural and elegant in this
context, exploiting the natural nested structure of the models and
exchangeability of the mixture components. As a result we remove the
need for calculation of a complicated Jacobian, reducing the potential
for making algebraic errors

It's a bit more complicated than your setting -- the full conditional for $k$ isn't available so $k$ is modified by a birth-death process
A: The alternative to reversible jump is to go for the respective posterior probabilities of the different models,
$$\mathbb P(\mathfrak M_k|\mathbf x)\propto
\pi(\mathfrak M_k) \underbrace{\int_{\Theta_k} \pi_k(\theta_k)f_k(\mathbf x|\theta_k)\,\text d\theta_k}_\text{$\mathfrak e_k$: evidence on $\mathfrak M_k$}$$
which can be approximated from simulations of $\pi_k(\theta_k|\mathbf x)$. Hence a marginal MCMC scheme could proceed by exploring the set of models (or model indices) as follows:
At iteration $t$, with current model index $k^{(t)}$

*

*make a proposal $k^\prime\sim q(\cdot|k^{(t)})$

*compute an unbiased estimator $\hat{\mathfrak e}_{k^\prime}$, e.g. by running some MCMC simulations targetting $\pi_{k^\prime}(\cdot|\mathbf x)$

*accept the move with probability $$\dfrac{\pi(\mathfrak M_{k^\prime})\hat{\mathfrak e}_{k^\prime}}{\pi(\mathfrak M_{k^{(t)}})\hat{\mathfrak e}_{k^{(t)}}}\,\dfrac{q(k^{(t)}|k^\prime)}{q(k^\prime|k^{(t)})}$$
In general, it is not possible to use solely simulations from the encompassing (or largest) model, since the posterior measures on the $\theta_k$'s are supported by sets of measure zero wrt the encompassing measure. This is why, even if model $\mathfrak M_k$ corresponds to a subset of $(\theta_1,\dots,\theta_{k_\text{max}}$ it must be treated as a completely different object, with a separate prior density.
