The Question is based on the paper titled : Image reconstruction in diffuse optical tomography using the coupled radiative transport–diffusion model

Download link

The Authors apply EM algorithm with $l_1$ sparsity regularization of an unknown vector $\mu$ to estimate the pixels of an image. The model is given by

$$y=A\mu + e \tag{1}$$ The estimate is given in Eq(8) as

$$\hat{\mu} = \arg max {\ln p(y|\mu) + \gamma \ln p(\mu)} \tag{2}$$

In my case, I have considered $\mu$ to be a filter of length $L$ and $\mathbf{\mu}$ are $L \times 1$ vectors representing the filters. So,

The model can be rewritten as $$y(n) = \mathbf{\mu^T}a(n) + v(n) \tag{3}$$

Question : Problem formulation : ${\mu(n)}$ (n by 1) is the unobserved input and $\{e(n)\}$ is the zero mean with unknown variance $\sigma^2_e$ additive noise. The MLE solution will based on Expectation Maximization (EM).

In the paper Eq(19) is the $A$ function - the complete log-likelihood but for my case I do not understand how I can include the distribution of $A, \mu$ in the the complete log-likelihood expression.

What will be the complete log-likelihood using EM of $y$ including the prior distribution?

  • $\begingroup$ Do you actually want the log-likelihood or do you instead want the log-posterior? Only the latter will include the Laplacian prior. The former can just be obtained by taking the log of the likelihood, which it seems you've already written out $\endgroup$
    – user44764
    Mar 9, 2016 at 21:56
  • $\begingroup$ There are two expressions that I want - (1) One that will be used to find the Fisher Information Matrix and the (2) other will be the pdf of the complete data set that includes the hidden variable $Z$ and the observances which is the joint probability density of the observed data as a function of the parameter $\theta$. The pdf that I have written is applicable to MA model for blind estimation of $\theta$. But, how will it be different for the sparsity constraint = Laplacian prior so that Fisher Information Matrix from the partial derivatives of the log-likelihood can be found. $\endgroup$
    – SKM
    Mar 9, 2016 at 22:06
  • $\begingroup$ @Xi'an: I do not understand how to plug in the 3 pdf's which includes the prior in the formulation of the log-likelihood. I can work out the maximization which is to take the partial derivative and equate to zero. Could you please put up an answer with the likelihood expression explicitly written out. This will really help $\endgroup$
    – SKM
    Mar 21, 2016 at 19:07

2 Answers 2


If we consider the target as $$\arg\max_\theta L(\theta|x)\pi(\theta) = \arg\max_\theta \log L(\theta|x) + \log \pi(\theta)$$ the representation at the basis of EM is $$\log L(\theta|x) = \mathbb{E}[\log L(\theta|x,Z)|x,\theta⁰]-\mathbb{E}[\log q(Z|x,\theta)|x,\theta⁰]$$ for an arbitrary $\theta⁰$, because of the decomposition $$q(z|x,\theta)=f(x,z|\theta) \big/ g(x|\theta)$$ or $$g(x|\theta) = f(x,z|\theta) \big/ q(z|x,\theta)$$ which works for an arbitrary value of $z$ (since there is none on the lhs) and hence also works for any expectation in $Z$: $$\log g(x|\theta) = \log f(x,z|\theta) - \log q(z|x,\theta) = \mathbb{E}[\log f(x,Z|\theta) - \log q(Z|x,\theta)|x]$$ for any conditional distribution of $Z$ given $X=x$, for instance $q(z|x,\theta⁰)$. Therefore if we maximise in $\theta$ $$\mathbb{E}[\log L(\theta|x,Z)|x,\theta⁰]+ \log \pi(\theta)$$ with solution $\theta^1$ we have $$\mathbb{E}[\log L(\theta^1|x,Z)|x,\theta⁰]+ \log \pi(\theta^1)\ge\mathbb{E}[\log L(\theta⁰|x,Z)|x,\theta⁰]+ \log \pi(\theta⁰)$$ while $$\mathbb{E}[\log q(Z|x,\theta⁰)|x,\theta⁰]\ge\mathbb{E}[\log q(Z|x,\theta^1)|x,\theta⁰]$$ by the standard arguments of EM. Therefore, $$\mathbb{E}[\log L(\theta^1|x,Z)|x,\theta⁰]+ \log \pi(\theta^1)\ge\mathbb{E}[\log L(\theta⁰|x,Z)|x,\theta⁰]+ \log \pi(\theta⁰)$$ and using as an E step the target $$\mathbb{E}[\log L(\theta|x,Z)|x,\theta⁰]+ \log \pi(\theta)$$ leads to an increase in the posterior at each M step, meaning that the modified EM algorithm converges to a local MAP.

  • $\begingroup$ Thank you for your reply. Does $q()$ represent the pdf of $Z$? Could you please why there are 2 expectations with $E[log q(.)]$ getting subtracted in the Equation mentioned in the second line? $\endgroup$
    – SKM
    Mar 23, 2016 at 16:44
  • $\begingroup$ I added some explanations, but you should check in a textbook the derivation of the EM algorithm since this is standard material. $\endgroup$
    – Xi'an
    Mar 23, 2016 at 16:52

I don't think showing monotonic increasing log-posterior (or log likelihood for MLE) are sufficient for showing convergence to stationary point of the MAP estimate (or MLE). For example, the increments can become arbitrarily small. In the famous paper by Wu 1983, a sufficient condition for converging to stationary point of EM is differentiability in both arguments of the lower bound function.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.