I have two questions in Energy function of expectation propagation. 1. Seeing the video of [1] (slide 24 of the second part), Minka says that, the evidence is the following: $$ Z = \left( \int q(\mathbf{x}) d\mathbf{x} \right) ^{1-n} \prod_{i=1}^{n} \int \frac{f_i (\mathbf{x})}{\tilde{f}_i(\mathbf{x}) } q(\mathbf{x}) d\mathbf{x} \quad \text{ (Equation a)} $$ But in [2] he introduces the primal energy function as it is shown in equation (6) of [2]. $$ \sum_i \int_{\mathbf{x}} \hat{p}_i(\mathbf{x}) \log \frac{ \hat{p}_i(\mathbf{y}) }{ f_i(\mathbf{x}) p(\mathbf{x}) } d\mathbf{x}‎ - ‎(n-1) \int_\mathbf{x} q(\mathbf{x}) \log \frac{q(\mathbf{x})}{p(\mathbf{x})} d\mathbf{x} \quad \text{ (Equation b)} $$ with $$ \hat{p}_i = \frac{1}{Z} q^{\setminus i} f_i(\mathbf{x}) $$ As I understand, we are trying to minimze $-\log Z$ which must be the same as equation (6) of [2]: $$ \log Z = \sum_{i=1}^{n} \log \int \frac{f_i (\mathbf{x})}{\tilde{f}_i(\mathbf{x}) } q(\mathbf{x}) d\mathbf{x} - (n-1) \log \int q(\mathbf{x}) d\mathbf{x} \quad \text{ (Equation c)} $$ However, I don't see how these are the same (UPDATE: I mean (Equation b) and (Equation c) ). Any idea on this?

  1. At [3] after equation 64 it says that : "The scale $s_a$ that minimizes local $\alpha$-divergence is ... ". Let's say we are in the case of KL divergence, and want to find $s_a$ for $\alpha=1$ (correct?). Since $s_a$ is defined to normalize the distribution $\tilde{f}_a(\mathbf{x})$, thus: $$ s_a = \frac{1} {\int \tilde{f}_a(\mathbf{x}) } $$ (right?) and $$ \tilde{f}_a(\mathbf{x})^{new} = \frac{ \arg\min_{q \in \mathcal{Q}} KL \left( q(\mathbf{x}) \frac{f_a(\mathbf{x})}{ \tilde{f}_a(\mathbf{x}) } || q^{\prime} \right) }{ q(\mathbf{x}) \frac{1}{ \tilde{f}_a(\mathbf{x}) } } \quad \text{ (Equation d)} $$ The paper [3] says that (for $\alpha=1$): $$ s_a = \frac{ \int q(\mathbf{x}) \frac{f_a(\mathbf{x})}{ \tilde{f}_a(\mathbf{x}) } d\mathbf{x} }{ \int q(\mathbf{x}) d\mathbf{x} } \quad \text{ (Equation e)} $$ However I don't see why. Any hints?

[1] videolectures.net/mlss09uk_minka_ai/

[2] http://research.microsoft.com/en-us/um/people/minka/papers/ep/minka-ep-energy.pdf

[3] http://www.seas.harvard.edu/courses/cs281/papers/minka-divergence.pdf


1 Answer 1

  1. The two functions aren't the same. They are only related by duality.
  2. $s_a$ is not a normalizer, it is an arbitrary scale factor.
  • $\begingroup$ "Minka says." :) $\endgroup$
    – David Marx
    Dec 2, 2013 at 7:02
  • $\begingroup$ Thank you for your answer. (1) I didn't mean the connection between primal and dual energy function. What I want is the way PRIMAL is derived. (Equation b) is the primal, and we know evidence is (Equation a), which is exactly the same as (Equation c). Now the question is how (Equation b) is related to (Equation c)? (2) Oh, I misunderstood. Then what equation do you solve to find $s_a$? In other words, what do you exactly mean by "the scale factor which minimizes the KL-divergence"? $\endgroup$
    – Asjai
    Dec 2, 2013 at 7:03
  • $\begingroup$ @TomMinka any chance you could provide a sketch of the derivation? I posted a question here: stats.stackexchange.com/questions/446930/… $\endgroup$ Jan 29, 2020 at 15:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.