My question is about what determines how hard it is to recover the number of components $K$ in a Gaussian mixture model (GMM), e.g. with the EM-algorithm.
For simplicity, let's consider the case in which the data generating GMM has $K=2$ components. It seems intuitive to me that the major driver of recovery-performance is how much the density between the two components is overlapping. Using the Kullball Keibler (KL) divergence seemed to be a good measure for this, because it incorporates both the means and the variances/covariances and is easy to calculate for multivariate Gaussian distributions (see e.g. here).
When varying the KL divergence and estimating $K$ it is indeed the case that for higher KL-divergences, the estimation performance is higher, which makes sense.
However, I then also considered cases in which I chose different arrangements of the mean vectors of the two Gaussian components, while keeping the KL divergence constant. To my surprise, I found that the specifications of the mixture components mattered for $K$-recovery beyond the KL divergence. Specifically, I found that the more the divergence is due to differences in means across many dimensions, the higher the performance. If the divergence is due to a difference only in the means of a single dimension, the performance is lowest. All of the above is about isomorphic Gaussians (diagonal covariance matrices with equal variances). Also note that this question only makes sense for $d>1$ dimensions, and is only true for $d>2$ dimensions (for $d=2$ it is easily verified geometrically that the mean structure doesn't matter for a constant KL divergence).
Can anyone point me towards an explanation for this? Intuitively, I thought that overlap between distribution should fully capture how difficult it is to estimate $K$, however, this does not seem to be the case.