Why is the E-step in the EM algorithm called this way?

Question

Yes, this has been asked before here, but for different reasons. In the E-Step nothing is calculated, we simply define the function, yet once it is defined it is defined once and for all. We could even define it in a way, such that it would not need to be redefined in each iteration by accepting a second parameter. It makes it look like the EM algorithm is a 2 step algorithm, while all it does it maximises the expectation in each iteration. I ask this question because I am not sure if I misunderstood something or if the description and presentation of the algorithm really is that bad or if there is a historical (or other) reason for the chosen description.

Answer 1

Let's use an example of a Gaussian mixture model. The model describes the distribution of your data in terms of $k$ clusters, such that each cluster is normally distributed. The model is

$$ f(x; \mu, \sigma,\pi) = \sum_{i=1}^K \pi_i\,\mathcal{N}(x\mid\mu_i,\sigma_i) $$

where $\mu = (\mu_1, \mu_2, \dots, \mu_K)$ is the vector or means, $\sigma = (\sigma_1, \sigma_2, \dots, \sigma_K)$ is the vector of variances, and $\pi = (\pi_1, \pi_2, \dots, \pi_K)$ is the vector of mixing proportions such that $\forall_i\,\pi_i \ge 0$ and $\sum_{i=1}^K \pi_i = 1$.

When we want to estimate the parameters of such a model, there is a chicken-and-egg problem: we could easily estimate the $\mu_i, \sigma_i,\pi_i$ per each cluster only if we knew to which cluster each observation belongs, and we could find out to which cluster the observation belongs if we knew the parameters.

Here E-M algorithm comes to play. You start with randomly initialized parameters and then iterate between the two steps. In the E step, you use the parameters to find the probabilities for each of the samples belonging to specific clusters. In the M step, you use those probabilities as weights to estimate and update the parameters. Such iterations are repeated until you don't see that they are changing anymore. In each step, you calculate something that wouldn't be possible to calculate without taking the other step.