3
$\begingroup$

Consider a Gaussian mixture of 2 components and a dataset of size $N$. The EM algorithm use the data to estimate:

  • the model parameters: the means $\mu_1, \mu_2$ (say the covariances matrices are fixed)
  • the conditional probability distributions of the latent variables: $p(z_i|x_i; \mu_1, \mu_2)$ for $i=1 .. N$

Because of the conditional probability distributions, the number of quantities we have to estimate grows linearly with $N$. How is EM not prone to overfitting?

My understanding is that it's because the $p(z_i|x_i; \mu_1, \mu_2)$ don't have any impact on the model complexity (whether a data point belongs to component 1 or 2 doesn't make the model more complex). Therefore, it is ok to have $N$ such distributions to estimate.

Is my understanding correct? Is this also true for other models using latent variables? i.e., are they usually used in a way that doesn't influence model complexity?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.