I have some difficulties understanding the phrase 'EM is a partially non-Bayesian method'. EM works in iterative fashion. Is it because the iterative nature of EM is somehow similar to prior - posterior relation in Bayesian network?

  • 4
    $\begingroup$ Could you provide a reference for the quote? $\endgroup$ – Xi'an Dec 1 '20 at 11:35
  • $\begingroup$ This might be the source of the quite: en.wikipedia.org/wiki/… $\endgroup$ – dwolfeu Dec 18 '20 at 15:17

EM is based on a demarginalisation of the (standard or observed) likelihood $$L^\text{o}(\theta|\mathbf x)=\int_{\mathfrak Z} L^\text{c}(\theta|\mathbf x,\mathbf z)\,\text d\mathbf z \tag{1}$$ introducing a latent variable $\mathbf Z$ to simplify the representation of the (observed) likelihood$$L^\text{o}(\theta|\mathbf x)$$into the completed likelihood$$L^\text{c}(\theta|\mathbf x,\mathbf z)$$but requiring (pseudo) inference on $\mathbf Z$ on the side. This inference is somewhat Bayesian in the sense that it uses the conditional distribution of $\mathbf Z$ given $\mathbf X=\mathbf x$ and the (current value of the) parameter $\theta$. Indeed, in the E step of the EM algorithm, a conditional expected log-likelihood is computed $$Q(\theta^{(t)},\theta|\mathbf x) = \mathbb E_{\theta^{(t)}} [\log L^\text{c}(\theta|\mathbf x,\mathbf Z) |\mathbf x ] \tag{2}$$ where the conditional expectation is against the conditional distribution of $\mathbf Z$ given the observation $\mathbf X=\mathbf x$ and $\theta=\theta^{(t)}$. However, the setting is not Bayesian in that

  1. while somewhat free, the "prior" distribution on $\mathbf Z$ is constrained by (1)
  2. there is no prior distribution on $\theta$ for EM and $\theta$ is never considered as a random variable by the EM algorithm
  3. EM results in finding a local mode of the observed likelihood, free from any prior input, and does not produce an inference on $\mathbf Z$

Another analogy can be found with Gibbs sampling, or more specifically data augmentation (Tanner & Wong, 1988) in that one iteration of Gibbs sampling looks like one iteration of the EM algorithm

  1. simulate $\mathbf z^{(t)}$ from $f(\cdot|\mathbf x,\theta^{(t)}$ versus compute (2) under $f(\cdot|\mathbf x,\theta^{(t)}$, which often results in computing $\mathbb E[\mathbf z^{(t)}|\mathbf x,\theta^{(t)}]$ (or even simulating $\mathbf z^{(t)}$ from $f(\cdot|\mathbf x,\theta^{(t)}$ in the MCEM version of Celeux and Diebolt (1980));

  2. simulate $\theta^{(t+1)}$ from $\pi(\cdot|\mathbf x,\theta^{(t)}$ versus maximise (2) in $\theta$ which results in $\theta^{(t+1)}$

As a last remark, let me point out that EM can be used in theory to find a MAP estimator associated with a prior density$\pi(\theta)$ by switching from $Q(\theta^{(t)},\theta|\mathbf x)$ in (2) to $$Q(\theta^{(t)},\theta|\mathbf x)+\log \pi(\theta)$$ in the E step, to be maximised in the M step. (The argument showing that EM increases the target at each iteration also applies there.)


In full Bayesian inference, you go from a prior distribution over parameter values, $P(\theta)$ to a posterior distribution given the data $P(\theta | x)$. With Expectation Maximisation, you go from a prior distribution to an estimate $\hat \theta$ of the most likely posterior value of $\theta$ given the data and the priors (the Maximum a Posteriori or MAP value). Since this is a point estimate, and not a full posterior distribution, it's not a fully Bayesian method.

  • $\begingroup$ It's hard for me to see how this answer maps onto the usual EM procedure. Could you clarify? $\endgroup$ – eric_kernfeld Dec 1 '20 at 20:37
  • $\begingroup$ But it's a useful complement to Xi'an's answer because it mentions the use of EM for MAP estimation, which Xi'an omits. $\endgroup$ – eric_kernfeld Dec 1 '20 at 20:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.