2
$\begingroup$

I am self learning Variational Inference. Currently I am reading the chapter 21 book from Murphy 1 and trying to understand the Ising model (21.3.2). The Ising model here is used as denoising technique to turn a gray valued image into a binary image.

The model is described as follows: assuming a latent variable $x_i \in \{-1, +1\}$, the joint model form is $p(\textbf{x}, \textbf{y}) = p(\textbf{x})p(\textbf{y}|\textbf{x})$, where $\textbf{y}$ denotes the gray values of the image and $\textbf{x}$ denotes the binary values of the corresponding pixels. The prior is $$p(\textbf{x}) \propto \exp(E_0(\textbf{x})),$$ $$E_0(\textbf{x}) = -\sum_{i=1}^{D}\sum_{j\in Nb(i)}W_{i,j}x_ix_j,$$ where $W$ is a weighting matrix (usually positive), and $Nb(i)$ denotes the first-degree hidden neighbors of pixel $i$.

Then, the author goes on and defines the likelihood as $$p(\textbf{y}|\textbf{x}) = \prod_{i}p(y_i|x_i) = \exp(\sum_{i}-L_i(x_i)).$$ Finally, the posterior is presented as $$p(\textbf{x}|\textbf{y}) \propto \exp(-E(\textbf{x})),$$ $$E(\textbf{x}) = E_0(\textbf{x}) - \sum_{i}L(x_i).$$

I do not follow what $L(\cdot)$ here represents, how the author arrived to the likelihood in that form, and how the posterior is determined since it should be intractable.

$\endgroup$
3
$\begingroup$

I do not follow what $L(\cdot)$ here represents

$L_i$ represents the log-likelihood of observing grayscale value $y_i$ if the binary value is $x_i$:

$$L_i(x_i) = \log p(y_i|x_i)$$

You can easily verify that taking exponential of sum of log-likelihoods equals taking product of likelihoods.

How the posterior is determined since it should be intractable

There is nothing intractable about writing down the posterior formula. Computing the posterior is intractable: you cannot compute it without evaluating all the possible values of $\mathbf{x}$, which are needed to compute the normalizer $Z$ in (21.40). That is why we need to use the approximate variational inference.


This IPython notebook might also provide you more insight.

$\endgroup$
  • $\begingroup$ Thanks for your answer. I did a mistake in copying the likelihood, the sum goes over the exponent and not as I put it before. I have corrected it. $\endgroup$ – mgbacher Oct 16 '18 at 13:08
  • $\begingroup$ @mgbacher No, the sum is inside the exponential (check (21.39) in the book). I corrected that for you in the question text, but I see you reverted it again... $\endgroup$ – Jan Kukacka Oct 16 '18 at 13:54
  • $\begingroup$ @mgbacher Product is converted to sum of powers $\exp(a+b)=\exp(a) \exp (b)$. $\endgroup$ – Jan Kukacka Oct 16 '18 at 13:57
  • $\begingroup$ No problem. Just for the reference, I corrected the equation in the question text once again. $\endgroup$ – Jan Kukacka Oct 17 '18 at 6:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.