I have the following distribution $$ \begin{align} \boldsymbol \pi&\sim\text{Dirichlet}([1,\cdots 1]\in R^K)\\ \boldsymbol \theta&\sim P(\boldsymbol \theta) \\ \mathbf y&\sim \sum _{i=1}^K\pi_i\text{Joint}(\theta_k) \end{align} $$ This is a Bayesian Mixture Model specified by the mixing weights $\pi_i$, paramaters $\theta_i$, and number of components $K$.

After sampling 5 chains from an HMC, the samples for $\boldsymbol \pi\in R^{5\times K}$ is not identifiable. In other words, if there are three dominant values for $\pi_i,\pi_j,\pi_k$ when $K=5$, these three dominants would be jumbled in different combinations.

This would cause MCMC diagnostics like r hat to give wrong values. In fact, MCMC diagnostics would give a nearly uniform distribution as the mean of the HMC sampled $\boldsymbol \pi$.

How do I perform diagnostics? Do i simply run a single but very large number of chain samples and split these ?

For example, sample $1000$ samples using an HMC sampler and split these into five $200$ samples and pass these into a MCMC diagnostics function?

  • $\begingroup$ Have you looked up the 'label switching problem' for mixture models? $\endgroup$
    – fm361
    Commented Sep 4, 2023 at 7:25

1 Answer 1


This is only a partial answer, but in general you can go a long way by adding an order constraint to your model: enforcing $\theta_1 \lt \theta_2 \lt\dots\lt \theta_k$. This is trickier to do if $\theta$ is multidimensional (forum).

You can find more on this in the stan docs. This guide is also useful, but quite technical.

  • $\begingroup$ May I ask if you know any bijectors that will enforce that constant? I'll lookup the mentioned forums for the meantime. $\endgroup$
    – wd violet
    Commented Sep 4, 2023 at 9:48
  • $\begingroup$ I found a TFP Ascending bijector which I will then chain with a Softmax. I'll try placing them on top of the Dirichlet distribution. Thanks $\endgroup$
    – wd violet
    Commented Sep 4, 2023 at 10:27
  • $\begingroup$ I don't know how in Tensorflow, sorry. $\endgroup$
    – Eoin
    Commented Sep 4, 2023 at 11:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.