How to apply EM algorithm in case of mixture distribution?

Question

I am familiar with regression linear models, and EM algorithms. However, I do not get the idea of fitting the mixture of regression linear models using the EM algorithm. So, what I think about it is as follows:

1. fit the first linear regression model and then estimate the coefficient. Then, find the density of the fitted model. I am confused about this part, as there is no density for the linear regression!
2. Repeat the first step with the second regression model.
3. Run EM algorithm.

Is that correct? Could someone help me with an example and manual implementation, please? I knew that there are some R packages, but I would like to understand the implementation manually.

Answer 1

No, the procedure is wrong.

If you're familiar with the Gaussian mixture, think of the mixture of regression as a Gaussian mixture, while the mean $\mu_k$ now has been substituted by $x_i^\top\beta_k$. Since when we talk about regression, we are talking about the mean regression.

To manually implement the algorithm, you should know how to derive the E-step and the M-step based on clear concepts of observed log-likelihood, complete-date log-likelihood, and expected complete-date likelihood w.r.t. the conditional distribution of missing labels.

Here is a detailed tutorial on how that works in the context of a mixture of regressions: Fitting mixtures of linear regressions.

Hope that helps.