Fitting a finite mixture: choice of the distribution and model selection for the number of components

This is a question about finite mixture models (FMM).

We want to fit a dataset $D$ but

1. we are not 100% sure of which distributions we should use to create the mixture;
2. we do not how many clusters are in the data.

So this is a bit of a hybrid question: we have "goodness of fit" to approach 1, and "model selection" to approach 2. There are several answers about 1/2 separately (here and elsewhere), but I could not find both approached together

I thought to do this, and I am wondering whether this is a sensible approach:

• assume (for simplicity) that "candidate" random variables for our FMM have densities $f$ or $g$ (e.g., a Beta and Beta-Binomial). So we want FMMs of the form $$M_1 = \sum_{k=1}^{K_1} \pi_k\, f(D|\theta_k) \quad and \quad M_2 = \sum_{k=1}^{K_2} \pi_k\, g(D|\eta_k)$$ where $\theta_x$ or $\eta_x$ are the parameters of the corresponding distributions, and $K_i$ is the size of the mixture, i.e., the number of components;
• I would first fit the first FMM's parameters via ML using criteria such as BIC/ ICL, and so I would learn its best values for $K_1$ and $\theta_x$;
• I would do the same (independently) for the second FMM, learning best values for $K_2$ and $\eta_x$;
• I would compare the likelihood functions of these models, via a standard LRT.

I am not sure that LRT is generally correct, however, as $f$ and $g$ are not nested in general. It could be that the single components of the mixtures are -- i.e., $f$ is Beta, $g$ is Beta-Binomial -- but it is not true in general that the overall FMMs are nested.

• (+1) Interesting query! The likelihood ratio tests do not work well for mixtures because of degeneracies on some subspaces of the parameter space. Furthermore, as you mention, since the two models are not embedded, the validation of the LRT is questionable in any case. – Xi'an May 30 '18 at 7:51

1 Answer

If the elements of the mixture are either Betas or Beta-Binomial, I would suggest using a mixture of both, $$\sum_{k=1}^{K_1} \omega\pi_k\, f(\cdot|\theta_k) + \sum_{k=1}^{K_2} (1-\omega)\varpi_k\, g(\cdot|\eta_k)$$ and solving the estimation in a Bayesian manner, using as a prior on $\omega$ a distribution peaked at zero and one. And estimate $K_1$ and $K_2$ by either a reversible jump algorithm à la Richardson and Green (1997) or by a saturated scheme à la Rousseau and Mengersen (2012).

• Thank you very much, I did not think about mixing the mixtures. This is certainly a very elegant formulation, and for $w$ one could use a Beta prior (I guess) with shapes $\alpha=\beta=0.5$. However, I am not sure that for the problem that I am interested in this is the best approach: I want to use the posterior estimates of the clusters to carry out downstream analyses. Thus, if the posterior estimates over $w$ is not sharp at 0/1, I would get a contribution that is a combination of two mixture components (maybe peaked similarly). Anyway, the idea is very interesting for density estimation. – qubert May 30 '18 at 9:50