A mixture of finite mixture models seem to be an interesting Bayesian (?) approach to solving clustering with an unknown $k$ number of components. It seems though, unlike the mixture model with a Dirichlet process prior, using a symmetric Dirichlet prior means $k$ doesn't necessarily grow with the size of the dataset $n$.

Unfortunately, I don't really understand how they are defined, or much else about them. The definition I am working with is from the paper by Miller and Harrison (2017) is as follows:

We consider the following well-known model:

$$ K ∼ p_K, \text{where }p_K \text{ is a p.m.f. on } \{1, 2, \ldots \} \\ (\pi_1, \ldots , \pi_k) ∼ \mathrm{Dirichlet}_k(γ, \ldots , γ), \text{ given }K = k \\ Z_1, \ldots , Z_n ~ \pi, \text{ given } \pi \\ \theta_1, \ldots , \theta_k ∼ H, \text{ given } K = k\\ X_j ∼ f_{θ_{Z_j}} \text{ independently for }j = 1, \ldots , n, \text { given } θ_{1:K}, Z_{1:n}. $$

Here, $H$ is a prior or “base measure” on $\Theta ⊂ \mathbb{R}^\ell$, and ${f_\theta : \theta \in \Theta}$ is a family of probability densities with respect to a sigma-finite measure $\zeta$ on $\chi \subset \mathbb{R}^d$. (As usual, we give Θ and $\chi$ the Borel sigma-algebra, and assume $(x, \theta) \rightarrow f_\theta(x)$ is measurable.) We denote $x_{1:n} = (x_1, \ldots , x_n)$. Typically, $X_1, . . . , X_n$ would be observed, and all other variables would be hidden/latent. We refer to this as a mixture of finite mixtures (MFM) model.

I think I understand the first three lines of definitions here but the rest seems very general and opaque.

What is finite mixture of mixtures? What is the intuition here? If people will humour me, I wouldn't mind also knowing :

  1. Why is there all this measure-theoretic terminology in the definition?
  2. How do I fit such a model? What is reversible jump Markov chain Monte Carlo and why is it so strongly associated with this model?
  3. Why isn't this more commonly used as it seems to be a nice solution to the problem of clustering with unknown $k$?

Mixture models with a prior on the number of components, J. W. Miller and M. T. Harrison, Journal of the American Statistical Association (JASA), Vol. 0, 2017, pp. 1-17.


Here is my understanding,

we draw the index or indicator as follows to indicate which $\theta_j$ to use $$ Z_i \sim \text{Categorical}(\pi_1, ..., \pi_k), i=1,...,n $$ then we draw the parameters of each component of a mixture model as, $$ \theta_j \sim H, j = 1,...,K $$ then we draw the sample according to the selected parameter $\theta_{Z_i}$


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.