Consider a Gaussian mixture of 2 components and a dataset of size $N$. The EM algorithm use the data to estimate:
- the model parameters: the means $\mu_1, \mu_2$ (say the covariances matrices are fixed)
- the conditional probability distributions of the latent variables: $p(z_i|x_i; \mu_1, \mu_2)$ for $i=1 .. N$
Because of the conditional probability distributions, the number of quantities we have to estimate grows linearly with $N$. How is EM not prone to overfitting?
My understanding is that it's because the $p(z_i|x_i; \mu_1, \mu_2)$ don't have any impact on the model complexity (whether a data point belongs to component 1 or 2 doesn't make the model more complex). Therefore, it is ok to have $N$ such distributions to estimate.
Is my understanding correct? Is this also true for other models using latent variables? i.e., are they usually used in a way that doesn't influence model complexity?