Skip to main content
3 of 11
added 2 characters in body
tomka
  • 6.7k
  • 7
  • 40
  • 85

Does EM algorithm require us to know the joint (predictive) distribution of the latent variables $Z$ when $Z$ is two-dimensional?

In its general form the E-step of the EM algorithm finds the expectation

$$ \int \log[ p(Y,Z | \theta)] p(Z|Y,\theta) d Z$$

where $Y$ the data, $Z$ the latent variables, $\theta$ the current parameters, and $l(\theta|Y,Z) = p(Y,Z|\theta)$ the complete data likelihood.

My question is: does EM require us to know the joint conditional (predictive) distribution of the latent variables, $p(Z|Y,\theta)$?

To illustrate. Now suppose $Z=[X,S]$ has two random variables parameterized by distinct vectors $\lambda$ and $\phi$ respectively, i.e. we know (can evaluate and sample from) $p(X|\lambda)$ and $p(S|\phi)$. $X,S$ are marginally independent; however they are not independent conditional on $Y$. We also know the complete data likelihood. Also we know the full conditionals $p(X|S,Y,\lambda)$ and $p(S|X,Y,\phi)$. If we fill this in we find:

$$ \int \log[ p(Y,X,S | \phi,\lambda)] p(X,S|Y, \phi,\lambda) d Z$$

Now the joint predictive distribution $p(X,S|Y, \phi,\lambda)$ is not known. We can write

$$ p(X|S,Y, \lambda) p(S|Y,\phi) = p(S|X,Y, \phi) p(X|Y,\lambda)$$

However the marginals $p(X|Y,\lambda),p(S|Y,\phi)$ are unknown.

Is there still a way to use a type of EM algorithm to solve this estimation problem?

tomka
  • 6.7k
  • 7
  • 40
  • 85