Variational Inference: Ising Model

Question

I am self learning Variational Inference. Currently I am reading the chapter 21 book from Murphy 1 and trying to understand the Ising model (21.3.2). The Ising model here is used as denoising technique to turn a gray valued image into a binary image.

The model is described as follows: assuming a latent variable $x_i \in \{-1, +1\}$, the joint model form is $p(\textbf{x}, \textbf{y}) = p(\textbf{x})p(\textbf{y}|\textbf{x})$, where $\textbf{y}$ denotes the gray values of the image and $\textbf{x}$ denotes the binary values of the corresponding pixels. The prior is $$p(\textbf{x}) \propto \exp(E_0(\textbf{x})),$$ $$E_0(\textbf{x}) = -\sum_{i=1}^{D}\sum_{j\in Nb(i)}W_{i,j}x_ix_j,$$ where $W$ is a weighting matrix (usually positive), and $Nb(i)$ denotes the first-degree hidden neighbors of pixel $i$.

Then, the author goes on and defines the likelihood as $$p(\textbf{y}|\textbf{x}) = \prod_{i}p(y_i|x_i) = \exp(\sum_{i}-L_i(x_i)).$$ Finally, the posterior is presented as $$p(\textbf{x}|\textbf{y}) \propto \exp(-E(\textbf{x})),$$ $$E(\textbf{x}) = E_0(\textbf{x}) - \sum_{i}L(x_i).$$

I do not follow what $L(\cdot)$ here represents, how the author arrived to the likelihood in that form, and how the posterior is determined since it should be intractable.

Answer 1

I do not follow what $L(\cdot)$ here represents

$L_i$ represents the log-likelihood of observing grayscale value $y_i$ if the binary value is $x_i$:

$$L_i(x_i) = \log p(y_i|x_i)$$

You can easily verify that taking exponential of sum of log-likelihoods equals taking product of likelihoods.

How the posterior is determined since it should be intractable

There is nothing intractable about writing down the posterior formula. Computing the posterior is intractable: you cannot compute it without evaluating all the possible values of $\mathbf{x}$, which are needed to compute the normalizer $Z$ in (21.40). That is why we need to use the approximate variational inference.

---

This IPython notebook might also provide you more insight.

Answer 2

The notation for this section is pretty bad. The $L(.)$ is first used to refer to the objective function to be maximized (ELBO). Then, as the accepted answer states, it is all of a sudden used for the log-likelihood.

It also doesn't help that the formula in the book (at least in my version, and it seems the original poster's as well) is incorrect and has the equation as:

$$ p(\textbf{y}|\textbf{x}) = \prod_{i}p(y_i|x_i) = \sum_{i}\exp(-L_i(x_i)). $$