How to diagnose HMC results like r-hat for a Mixture Model?

Question

I have the following distribution $$ \begin{align} \boldsymbol \pi&\sim\text{Dirichlet}([1,\cdots 1]\in R^K)\\ \boldsymbol \theta&\sim P(\boldsymbol \theta) \\ \mathbf y&\sim \sum _{i=1}^K\pi_i\text{Joint}(\theta_k) \end{align} $$ This is a Bayesian Mixture Model specified by the mixing weights $\pi_i$, paramaters $\theta_i$, and number of components $K$.

After sampling 5 chains from an HMC, the samples for $\boldsymbol \pi\in R^{5\times K}$ is not identifiable. In other words, if there are three dominant values for $\pi_i,\pi_j,\pi_k$ when $K=5$, these three dominants would be jumbled in different combinations.

This would cause MCMC diagnostics like r hat to give wrong values. In fact, MCMC diagnostics would give a nearly uniform distribution as the mean of the HMC sampled $\boldsymbol \pi$.

---

How do I perform diagnostics? Do i simply run a single but very large number of chain samples and split these ?

For example, sample $1000$ samples using an HMC sampler and split these into five $200$ samples and pass these into a MCMC diagnostics function?

Answer 1

This is only a partial answer, but in general you can go a long way by adding an order constraint to your model: enforcing $\theta_1 \lt \theta_2 \lt\dots\lt \theta_k$. This is trickier to do if $\theta$ is multidimensional (forum).

You can find more on this in the stan docs. This guide is also useful, but quite technical.