Consider a Gaussian mixture of 2 components and a dataset of size $N$. The EM algorithm use the data to estimate:

  • the model parameters: the means $\mu_1, \mu_2$ (say the covariances matrices are fixed)
  • the conditional probability distributions of the latent variables: $p(z_i|x_i; \mu_1, \mu_2)$ for $i=1 .. N$

Because of the conditional probability distributions, the number of quantities we have to estimate grows linearly with $N$. How is EM not prone to overfitting?

My understanding is that it's because the $p(z_i|x_i; \mu_1, \mu_2)$ don't have any impact on the model complexity (whether a data point belongs to component 1 or 2 doesn't make the model more complex). Therefore, it is ok to have $N$ such distributions to estimate.

Is my understanding correct? Is this also true for other models using latent variables? i.e., are they usually used in a way that doesn't influence model complexity?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.