2
$\begingroup$

There is a PDF $p(D| \theta)=p(X,Z| \theta)$ with observed values $X$ but also some missing or incomplete values $Z$ (for eg. resulting from censoring).

The expectation-maximization (EM) algorithm is said to give the parameters $\hat{\theta}$ that maximize the likelihood of $D$--i.e. for both the observed and missing values. Call this "property 1."

The EM algorithm is also used to estimate the weights $w_k$ and parameters $\theta_k$ of a mixture $M(x)$ of $K$ component PDFs $p_c(x| \theta_k)$:

$$M(x)=\sum_{k=1}^{K} w_{k} p_c(x| \theta_k) $$

In this case, the probability that an observation $x_i$ is generated by the PDF component $k$ is treated as the missing values $Z$.

My question is does using the EM algorithm to solve for the weights and parameters of the mixture $M$ that maximize the log likelihood also have "property 1?" I.e. does it maximize likelihood for the whole distribution of both observed and missing values?

To formulate my question I am drawing heavily on this document:

https://courses.csail.mit.edu/6.867/wiki/images/b/b5/Em_tutorial.pdf

$\endgroup$
1
$\begingroup$

Let $X$ be the observed data and $Z$ be the latent data. The EM algorithm is an algorithm to maximize $p(X \mid \theta)$. When you say

The expectation-maximization (EM) algorithm is said to give the parameters $\hat \theta$ that maximize the likelihood of $D$, i.e. for both the observed and missing values. Call this "property 1."

it sounds to me like property 1 is saying the EM algorithm maximizes the likeihood of $p(X, Z \mid \theta)$, as a function of $\theta$. This is impossible - we don't know $Z$, hence there is no way to compute $\hat \theta(X, Z) = \arg\max_\theta p(X, Z \mid \theta)$. What the EM algorithm does is (attempt to) calculate $\hat \theta(X) = \arg \max_\theta \int p(X, Z \mid \theta) \ dZ$, but there is no reason to expect that $\hat \theta(X) = \hat \theta(X, Z)$.

$\endgroup$
-1
$\begingroup$

I believe the answer is YES. The mixture problem is one of the famous example of EM algorithm, e.g., Gaussian mixture. BTW, the link in the question sees to be dead.

As a meta-algorithm, EM algorithm directly maximizes the expected complete data likelihood, but can guarantee the increase of observed data likelihood. The correct version of the proof is in the following reference.

Wu, C. F. Jeff (1983). "On the Convergence Properties of the EM Algorithm". Annals of Statistics 11 (1): 95–103.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.