2
$\begingroup$

I've seen this paper's straightforward description of the EM algorithm cited countless times now to explain EM (figure below). But it's only causing me more confusion because I have trouble seeing how it aligns with EM theory.

In theory, the M step of EM is about finding the $\theta$ values that maximize the expectation of the log likelihood, given the current guess of parameters. However, this paper does not seem to calculate the expectation of the log likelihood at all. Instead, it multiplies the heads & tails counts with normalized likelihoods of coin A and coin B.

Am I missing something? Is it proven somewhere that for this particular model, this procedure maximizes the expectation of the log likelihood, and it's just not shown in the paper? I'm having a lot of trouble reconciling the EM algorithm with this very straightforward example.

enter image description here

$\endgroup$
2
$\begingroup$

By updating the $\theta$s according to those formula, you are already maximising the expectation of the log-likelihood functions. In other words, if you write down the expectation of the log-likelihood function (w.r.t the hidden variables), take its derivatives w.r.t the parameters, set them to zero, and solve for the $\theta$s, you will end up with those formulas. I believe the article was trying to make the exposition as introductory and intuitive as possible so it did not show this technical process, thus your confusion.

It should be noted that the EM algorithm tells you what to maximise (i.e. the expectation of the log-likelihoods), not how to maximise (i.e. the exact formula for updating the parameters); it is therefore not a specific algorithm, but rather a "template algorithm" for deriving solutions that are specific to each problem. As the practitioner of the EM algorithm, it is your responsibility to derive the formula that would maximise the expectation function (often denoted as the $Q$ function in the literature).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.