Why do we say EM is a partially non-Bayesian method?

Question

I have some difficulties understanding the phrase 'EM is a partially non-Bayesian method'. EM works in iterative fashion. Is it because the iterative nature of EM is somehow similar to prior - posterior relation in Bayesian network?

Answer 1

EM is based on a demarginalisation of the (standard or observed) likelihood $$L^\text{o}(\theta|\mathbf x)=\int_{\mathfrak Z} L^\text{c}(\theta|\mathbf x,\mathbf z)\,\text d\mathbf z \tag{1}$$ introducing a latent variable $\mathbf Z$ to simplify the representation of the (observed) likelihood$$L^\text{o}(\theta|\mathbf x)$$into the completed likelihood$$L^\text{c}(\theta|\mathbf x,\mathbf z)$$but requiring (pseudo) inference on $\mathbf Z$ on the side. This inference is somewhat Bayesian in the sense that it uses the conditional distribution of $\mathbf Z$ given $\mathbf X=\mathbf x$ and the (current value of the) parameter $\theta$. Indeed, in the E step of the EM algorithm, a conditional expected log-likelihood is computed $$Q(\theta^{(t)},\theta|\mathbf x) = \mathbb E_{\theta^{(t)}} [\log L^\text{c}(\theta|\mathbf x,\mathbf Z) |\mathbf x ] \tag{2}$$ where the conditional expectation is against the conditional distribution of $\mathbf Z$ given the observation $\mathbf X=\mathbf x$ and $\theta=\theta^{(t)}$. However, the setting is not Bayesian in that

1. while somewhat free, the "prior" distribution on $\mathbf Z$ is constrained by (1)
2. there is no prior distribution on $\theta$ for EM and $\theta$ is never considered as a random variable by the EM algorithm
3. EM results in finding a local mode of the observed likelihood, free from any prior input, and does not produce an inference on $\mathbf Z$

Another analogy can be found with Gibbs sampling, or more specifically data augmentation (Tanner & Wong, 1988) in that one iteration of Gibbs sampling looks like one iteration of the EM algorithm

1. simulate $\mathbf z^{(t)}$ from $f(\cdot|\mathbf x,\theta^{(t)}$ versus compute (2) under $f(\cdot|\mathbf x,\theta^{(t)}$, which often results in computing $\mathbb E[\mathbf z^{(t)}|\mathbf x,\theta^{(t)}]$ (or even simulating $\mathbf z^{(t)}$ from $f(\cdot|\mathbf x,\theta^{(t)}$ in the MCEM version of Celeux and Diebolt (1980));

2. simulate $\theta^{(t+1)}$ from $\pi(\cdot|\mathbf x,\theta^{(t)}$ versus maximise (2) in $\theta$ which results in $\theta^{(t+1)}$

As a last remark, let me point out that EM can be used in theory to find a MAP estimator associated with a prior density$\pi(\theta)$ by switching from $Q(\theta^{(t)},\theta|\mathbf x)$ in (2) to $$Q(\theta^{(t)},\theta|\mathbf x)+\log \pi(\theta)$$ in the E step, to be maximised in the M step. (The argument showing that EM increases the target at each iteration also applies there.)

Answer 2

In full Bayesian inference, you go from a prior distribution over parameter values, $P(\theta)$ to a posterior distribution given the data $P(\theta | x)$. With Expectation Maximisation, you go from a prior distribution to an estimate $\hat \theta$ of the most likely posterior value of $\theta$ given the data and the priors (the Maximum a Posteriori or MAP value). Since this is a point estimate, and not a full posterior distribution, it's not a fully Bayesian method.