I am self learning Variational Inference. Currently I am reading the chapter 21 book from Murphy 1 and trying to understand the Ising model (21.3.2). The Ising model here is used as denoising technique to turn a gray valued image into a binary image.

The model is described as follows: assuming a latent variable $x_i \in \{-1, +1\}$, the joint model form is $p(\textbf{x}, \textbf{y}) = p(\textbf{x})p(\textbf{y}|\textbf{x})$, where $\textbf{y}$ denotes the gray values of the image and $\textbf{x}$ denotes the binary values of the corresponding pixels. The prior is $$p(\textbf{x}) \propto \exp(E_0(\textbf{x})),$$ $$E_0(\textbf{x}) = -\sum_{i=1}^{D}\sum_{j\in Nb(i)}W_{i,j}x_ix_j,$$ where $W$ is a weighting matrix (usually positive), and $Nb(i)$ denotes the first-degree hidden neighbors of pixel $i$.

Then, the author goes on and defines the likelihood as $$p(\textbf{y}|\textbf{x}) = \prod_{i}p(y_i|x_i) = \exp(\sum_{i}-L_i(x_i)).$$ Finally, the posterior is presented as $$p(\textbf{x}|\textbf{y}) \propto \exp(-E(\textbf{x})),$$ $$E(\textbf{x}) = E_0(\textbf{x}) - \sum_{i}L(x_i).$$

I do not follow what $L(\cdot)$ here represents, how the author arrived to the likelihood in that form, and how the posterior is determined since it should be intractable.


2 Answers 2


I do not follow what $L(\cdot)$ here represents

$L_i$ represents the log-likelihood of observing grayscale value $y_i$ if the binary value is $x_i$:

$$L_i(x_i) = \log p(y_i|x_i)$$

You can easily verify that taking exponential of sum of log-likelihoods equals taking product of likelihoods.

How the posterior is determined since it should be intractable

There is nothing intractable about writing down the posterior formula. Computing the posterior is intractable: you cannot compute it without evaluating all the possible values of $\mathbf{x}$, which are needed to compute the normalizer $Z$ in (21.40). That is why we need to use the approximate variational inference.

This IPython notebook might also provide you more insight.

  • $\begingroup$ Thanks for your answer. I did a mistake in copying the likelihood, the sum goes over the exponent and not as I put it before. I have corrected it. $\endgroup$
    – mgbacher
    Commented Oct 16, 2018 at 13:08
  • $\begingroup$ @mgbacher No, the sum is inside the exponential (check (21.39) in the book). I corrected that for you in the question text, but I see you reverted it again... $\endgroup$ Commented Oct 16, 2018 at 13:54
  • $\begingroup$ @mgbacher Product is converted to sum of powers $\exp(a+b)=\exp(a) \exp (b)$. $\endgroup$ Commented Oct 16, 2018 at 13:57
  • $\begingroup$ No problem. Just for the reference, I corrected the equation in the question text once again. $\endgroup$ Commented Oct 17, 2018 at 6:30

The notation for this section is pretty bad. The $L(.)$ is first used to refer to the objective function to be maximized (ELBO). Then, as the accepted answer states, it is all of a sudden used for the log-likelihood.

It also doesn't help that the formula in the book (at least in my version, and it seems the original poster's as well) is incorrect and has the equation as:

$$ p(\textbf{y}|\textbf{x}) = \prod_{i}p(y_i|x_i) = \sum_{i}\exp(-L_i(x_i)). $$


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.