# Why do we say EM is a partially non-Bayesian method?

I have some difficulties understanding the phrase 'EM is a partially non-Bayesian method'. EM works in iterative fashion. Is it because the iterative nature of EM is somehow similar to prior - posterior relation in Bayesian network?

• Could you provide a reference for the quote? – Xi'an Dec 1 '20 at 11:35
• This might be the source of the quite: en.wikipedia.org/wiki/… – dwolfeu Dec 18 '20 at 15:17

EM is based on a demarginalisation of the (standard or observed) likelihood $$L^\text{o}(\theta|\mathbf x)=\int_{\mathfrak Z} L^\text{c}(\theta|\mathbf x,\mathbf z)\,\text d\mathbf z \tag{1}$$ introducing a latent variable $$\mathbf Z$$ to simplify the representation of the (observed) likelihood$$L^\text{o}(\theta|\mathbf x)$$into the completed likelihood$$L^\text{c}(\theta|\mathbf x,\mathbf z)$$but requiring (pseudo) inference on $$\mathbf Z$$ on the side. This inference is somewhat Bayesian in the sense that it uses the conditional distribution of $$\mathbf Z$$ given $$\mathbf X=\mathbf x$$ and the (current value of the) parameter $$\theta$$. Indeed, in the E step of the EM algorithm, a conditional expected log-likelihood is computed $$Q(\theta^{(t)},\theta|\mathbf x) = \mathbb E_{\theta^{(t)}} [\log L^\text{c}(\theta|\mathbf x,\mathbf Z) |\mathbf x ] \tag{2}$$ where the conditional expectation is against the conditional distribution of $$\mathbf Z$$ given the observation $$\mathbf X=\mathbf x$$ and $$\theta=\theta^{(t)}$$. However, the setting is not Bayesian in that

1. while somewhat free, the "prior" distribution on $$\mathbf Z$$ is constrained by (1)
2. there is no prior distribution on $$\theta$$ for EM and $$\theta$$ is never considered as a random variable by the EM algorithm
3. EM results in finding a local mode of the observed likelihood, free from any prior input, and does not produce an inference on $$\mathbf Z$$

Another analogy can be found with Gibbs sampling, or more specifically data augmentation (Tanner & Wong, 1988) in that one iteration of Gibbs sampling looks like one iteration of the EM algorithm

1. simulate $$\mathbf z^{(t)}$$ from $$f(\cdot|\mathbf x,\theta^{(t)}$$ versus compute (2) under $$f(\cdot|\mathbf x,\theta^{(t)}$$, which often results in computing $$\mathbb E[\mathbf z^{(t)}|\mathbf x,\theta^{(t)}]$$ (or even simulating $$\mathbf z^{(t)}$$ from $$f(\cdot|\mathbf x,\theta^{(t)}$$ in the MCEM version of Celeux and Diebolt (1980));

2. simulate $$\theta^{(t+1)}$$ from $$\pi(\cdot|\mathbf x,\theta^{(t)}$$ versus maximise (2) in $$\theta$$ which results in $$\theta^{(t+1)}$$

As a last remark, let me point out that EM can be used in theory to find a MAP estimator associated with a prior density$$\pi(\theta)$$ by switching from $$Q(\theta^{(t)},\theta|\mathbf x)$$ in (2) to $$Q(\theta^{(t)},\theta|\mathbf x)+\log \pi(\theta)$$ in the E step, to be maximised in the M step. (The argument showing that EM increases the target at each iteration also applies there.)

In full Bayesian inference, you go from a prior distribution over parameter values, $$P(\theta)$$ to a posterior distribution given the data $$P(\theta | x)$$. With Expectation Maximisation, you go from a prior distribution to an estimate $$\hat \theta$$ of the most likely posterior value of $$\theta$$ given the data and the priors (the Maximum a Posteriori or MAP value). Since this is a point estimate, and not a full posterior distribution, it's not a fully Bayesian method.

• It's hard for me to see how this answer maps onto the usual EM procedure. Could you clarify? – eric_kernfeld Dec 1 '20 at 20:37
• But it's a useful complement to Xi'an's answer because it mentions the use of EM for MAP estimation, which Xi'an omits. – eric_kernfeld Dec 1 '20 at 20:38