Is there a relationship between the number of the mixture components and the overfiting of the model?

Question

I read the following:

To prevent overfitting we would like to work with as few components as possible".

How does the number of the mixture component affect the fit of the model? Is that because the selection criteria, such as AIC tend to select a model with more parameters? or there are other reasons? If yes, what about BIC? I think it provides a better choice than AIC. So, why the number of components may lead to an overfitting issue?

Answer 1

Yes. In general, when the model has more parameters, it is usually more expressive, so easier to overfit (fit to the noise).

Think of a one-dimensional example of a Gaussian mixture:

• with one dimension, you are fitting a Gaussian to the data, it can fit or not, for non-normally distributed data it will underfit,
• with two components, it can fit bimodal distributions or distributions with a certain kind of fatter tails ("narrow" Gaussian mixed with "flat" gaussian with large variance),
• with more components, it can approximate many shapes of distributions,
• with as many components as data points, it is a special case of a non-parametric Kernel density estimator with Gaussian kernels that can approximate the shape of any distribution and can easily overfit the data if you don't set the bandwidth to a reasonable value.