My question is about what determines how hard it is to recover the number of components $K$ in a Gaussian mixture model (GMM), e.g. with the EM-algorithm.

For simplicity, let's consider the case in which the data generating GMM has $K=2$ components. It seems intuitive to me that the major driver of recovery-performance is how much the density between the two components is overlapping. Using the Kullball Keibler (KL) divergence seemed to be a good measure for this, because it incorporates both the means and the variances/covariances and is easy to calculate for multivariate Gaussian distributions (see e.g. here).

When varying the KL divergence and estimating $K$ it is indeed the case that for higher KL-divergences, the estimation performance is higher, which makes sense.

However, I then also considered cases in which I chose different arrangements of the mean vectors of the two Gaussian components, while keeping the KL divergence constant. To my surprise, I found that the specifications of the mixture components mattered for $K$-recovery beyond the KL divergence. Specifically, I found that the more the divergence is due to differences in means across many dimensions, the higher the performance. If the divergence is due to a difference only in the means of a single dimension, the performance is lowest. All of the above is about isomorphic Gaussians (diagonal covariance matrices with equal variances). Also note that this question only makes sense for $d>1$ dimensions, and is only true for $d>2$ dimensions (for $d=2$ it is easily verified geometrically that the mean structure doesn't matter for a constant KL divergence).

Can anyone point me towards an explanation for this? Intuitively, I thought that overlap between distribution should fully capture how difficult it is to estimate $K$, however, this does not seem to be the case.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.