Assessing Gaussian mixture distribution by cross validation

Question

I have a 10 dimensional random vector that I'm modelling with GMMs. I want to estimate the best number of mixtures ($K$) for my data via the following method:

1. Divide the data to train (90%) and test (10%) by 10-fold cross-validation (i.e. perform 10 experiments where each time a different part is the test).
2. Fit GMMs with $K$ components.
3. Check the mean log pdf of the test data using the trained GMM. For each $K$, do that 10 times (for cross validation) and get the mean value.

Then, the $K$ that yields the maximal mean log-pdf value is the best $K$.

I always keep the number of samples per GMM parameters above $20$ (and even more) so that the EM algorithm will converge.

However, I'm getting an increasing mean log-pdf figure with my number of components. I even got to $1000$ components and got the maximal mean log-pdf on the test set. How is that possible for a 10-sized random vector? shouldn't it overfit at this point and therefore yield lower pdf values on the test set?

Thanks

EDIT (addition): I also tried calculating the AIC for my data: $$\mbox{AIC}=2p - 2\log \hat L, $$ which apparently should consider the log-likelihood ($\hat L$) versus the model complexity (expressed via number of parameters, $p$). Here as well, I see an increasing graph even for a very high number of components. I use the normalization method often used in the literature, $$r_k = \frac{ \exp\{-0.5(AIC_k - AIC_{min})\}} { \sum_{i=1}^K \exp\{-0.5(AIC_i - AIC_{min})\} }$$ that scales the AIC values, but then I see a value very close to $1$ at the highest number of components I use, and values very close to zero at the rest of the trials (for lower components).

I am not sure if there's a thing I'm missing here, or if I should just increase the number of components (above 1000 components, for a 10-sized vector). I would appreciate any insights on the matter.

Answer 1

Whit clustering it is very likely that the error measure decreases (even on test set) as you increase the number of clusters (here the number of components in GMM). There are two competing factors that change the error metric on the test set.

1. As you increase the number of clusters each cluster will have less data to be trained on. Hence the parameters of the cluster are obtained with less accuracy. This will increase the test error.
2. As you increase the number of clusters they tend to fill the feature space more densely. Hence it is more likely that the data points in test set would be close to one of the cluster centers. This will decrease the test error and is often the dominant factor.

For example consider the following picture. Modelling this data as mixture of two Gaussian distributions seems sufficient. But you will have some dispersion as not all the points are at the mean of each Gaussian.

If you continue and model it with a mixture of a higher number of Gaussian distributions then the total likelihood will increase on the training set. At the same time it is very likely that the likelihood on the test set increases too.

When you use AIC you add a penalty term $2p$ which is not enough in your case to change the direction of decreasing the dispersion on test set.

For determining the number of clusters generally people look at the elbow on the test error. When you add the AIC of BIC penalty then you increase the sharpness of the elbow.

https://en.wikipedia.org/wiki/Determining_the_number_of_clusters_in_a_data_set#cite_note-6

You can also have look at the 'elements of statistical learning' pages 518-520, available at: http://statweb.stanford.edu/~tibs/ElemStatLearn/printings/ESLII_print10.pdf