3
$\begingroup$

I have two questions in Energy function of expectation propagation. 1. Seeing the video of [1] (slide 24 of the second part), Minka says that, the evidence is the following: $$ Z = \left( \int q(\mathbf{x}) d\mathbf{x} \right) ^{1-n} \prod_{i=1}^{n} \int \frac{f_i (\mathbf{x})}{\tilde{f}_i(\mathbf{x}) } q(\mathbf{x}) d\mathbf{x} \quad \text{ (Equation a)} $$ But in [2] he introduces the primal energy function as it is shown in equation (6) of [2]. $$ \sum_i \int_{\mathbf{x}} \hat{p}_i(\mathbf{x}) \log \frac{ \hat{p}_i(\mathbf{y}) }{ f_i(\mathbf{x}) p(\mathbf{x}) } d\mathbf{x}‎ - ‎(n-1) \int_\mathbf{x} q(\mathbf{x}) \log \frac{q(\mathbf{x})}{p(\mathbf{x})} d\mathbf{x} \quad \text{ (Equation b)} $$ with $$ \hat{p}_i = \frac{1}{Z} q^{\setminus i} f_i(\mathbf{x}) $$ As I understand, we are trying to minimze $-\log Z$ which must be the same as equation (6) of [2]: $$ \log Z = \sum_{i=1}^{n} \log \int \frac{f_i (\mathbf{x})}{\tilde{f}_i(\mathbf{x}) } q(\mathbf{x}) d\mathbf{x} - (n-1) \log \int q(\mathbf{x}) d\mathbf{x} \quad \text{ (Equation c)} $$ However, I don't see how these are the same (UPDATE: I mean (Equation b) and (Equation c) ). Any idea on this?

  1. At [3] after equation 64 it says that : "The scale $s_a$ that minimizes local $\alpha$-divergence is ... ". Let's say we are in the case of KL divergence, and want to find $s_a$ for $\alpha=1$ (correct?). Since $s_a$ is defined to normalize the distribution $\tilde{f}_a(\mathbf{x})$, thus: $$ s_a = \frac{1} {\int \tilde{f}_a(\mathbf{x}) } $$ (right?) and $$ \tilde{f}_a(\mathbf{x})^{new} = \frac{ \arg\min_{q \in \mathcal{Q}} KL \left( q(\mathbf{x}) \frac{f_a(\mathbf{x})}{ \tilde{f}_a(\mathbf{x}) } || q^{\prime} \right) }{ q(\mathbf{x}) \frac{1}{ \tilde{f}_a(\mathbf{x}) } } \quad \text{ (Equation d)} $$ The paper [3] says that (for $\alpha=1$): $$ s_a = \frac{ \int q(\mathbf{x}) \frac{f_a(\mathbf{x})}{ \tilde{f}_a(\mathbf{x}) } d\mathbf{x} }{ \int q(\mathbf{x}) d\mathbf{x} } \quad \text{ (Equation e)} $$ However I don't see why. Any hints?

[1] videolectures.net/mlss09uk_minka_ai/

[2] http://research.microsoft.com/en-us/um/people/minka/papers/ep/minka-ep-energy.pdf

[3] http://www.seas.harvard.edu/courses/cs281/papers/minka-divergence.pdf

$\endgroup$
4
$\begingroup$
  1. The two functions aren't the same. They are only related by duality.
  2. $s_a$ is not a normalizer, it is an arbitrary scale factor.
$\endgroup$
  • $\begingroup$ "Minka says." :) $\endgroup$ – David Marx Dec 2 '13 at 7:02
  • $\begingroup$ Thank you for your answer. (1) I didn't mean the connection between primal and dual energy function. What I want is the way PRIMAL is derived. (Equation b) is the primal, and we know evidence is (Equation a), which is exactly the same as (Equation c). Now the question is how (Equation b) is related to (Equation c)? (2) Oh, I misunderstood. Then what equation do you solve to find $s_a$? In other words, what do you exactly mean by "the scale factor which minimizes the KL-divergence"? $\endgroup$ – Asjai Dec 2 '13 at 7:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.