# comparing gaussian mixture models with and without regularization

I am using Gaussian mixture models for clustering a bunch of data sets.

On some data sets a regularization value is often used/suggested (see the MATLAB example for Fisher's Iris data which sets RegularizationValue = 0.01).

I could not find a systematic way to set such a regularization value for a given data set.

My question is, is it valid to compare two Gaussian mixture models (e.g. using their negative log-likelihood or criterion like AIC/BIC) one of which uses a non-zero regularization value and another does not?

If I recall correctly, Matlab's GMM regularization works by adding a multiple of the identity matrix to the covariance matrices.

Using regularization can be viewed as placing a prior on the covariance matrices. As the strength of regularization increases, the model becomes less flexible. As a consequence, the training set log likelihood may be lower for more strongly regularized models. Because of this, training set log likelihood isn't a great way to compare models with different levels of regularization.

AIC and BIC penalize more flexible models, so this is a step in the right direction. They do this by measuring the number of free parameters, and are straightforward to use for comparing models with different number of mixture components and different types of covariance matrices (tied, diagonal, full, etc.). In these cases, the number of free parameters can simply be counted. The situation is more complicated for regularized covariance matrices, because model flexibility is governed by a continuous regularization parameter, rather than a change in the number of parameters. It may be possible to find an expression for the 'effective degrees of freedom' as a function of the regularization parameter. For example, this can be done in the case of ridge regression.

Two alternatives come to mind (I'm assuming you're holding the number of components, type of covariance matrix, etc. fixed, and only changing the regularization parameter). 1) Train each model on a subset of your data, then compute the log likelihood on the remaining test set (better yet, use cross validation). Choose the regularization parameter that maximizes test set likelihood. 2) Use Bayesian model comparison. Choose the regularization parameter that maximizes the evidence. This is a more complicated approach, because you'd have to define priors on all parameters, derive an expression for the evidence, and find a way to compute or approximate it.

One reason to prefer cross validation/Bayesian model comparison over AIC/BIC is that AIC/BIC rely on asymptotic approximations, which may or may not be appropriate for your data. Note that cross validation is not a valid strategy for choosing the number of mixture components, because the test set likelihood may increase monotonically with the number of components.

• thanks for the answer. I have two additional queries (perhaps they will qualify as a separate question). (1) Can we count regularization as reducing number of free parameters by some amount? (2) The MATLAB regularization (and any other similar tools) how is the regularization implemented, does it add it to the diagonal before every inversion?
– Krrr
Commented Jul 23, 2017 at 8:47
• @user20160 : you specifically mentioned that CV shall not be used to choose the number of mixture components. How would you suggest to choose the number of mixture compolents? Commented Feb 17, 2021 at 20:24