I was reading this tutorial related to EM algorithm at http://aass.oru.se/~tdt/ml/extra-readings/EM_algorithm.pdf. As given in the tutorial

enter image description here

we can see that at each E step we calculate the expectation of the likelihood over the posterior distribution of the hidden variables and them maximize it.The expected log likelihood over the posterior distribution hidden variables is upper bounded by the likelihood over the observed data only.

My questions are:

  • Why is $L(\theta)$ non convex? Isn't it possible for $L(\theta)$ to be convex?
  • Further why is $l(\theta | \theta_n)$ a convex function as shown in the figure?

The likelihood, likelihood function, or p(y|Θ), contains the available information provided by the sample. The data y affects the posterior distribution p(Θ|y) only through the likelihood function p(y|Θ). The likelihood function is what determines your model. For some very specific simple models, the likelihood function may indeed be convex, in which case the EM algorithm would collapse to a single step, and the inference would be exact. However for most models of interest, the likelihood function will be (sometimes highly) non-convex.

l(θ|θn) is the arg-max of the expectation of complete data log-likelihood using the hidden variables z. Although this function may not be strictly convex, the hidden variables are usually chosen so that it is convex, or otherwise easily optimised. The particular choices depend on the likelihood function.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.