7
$\begingroup$

When reading a presentation on "expectation propagation," I found a strange formula for computing $\text{E}[X]$ from a conditional probability:

$$\text{E}[X] = \frac{\int x P(y_i|x) dx}{\int P(y_i|x) dx}$$

Can someone please explain how this formula was derived? The source is slide 24 in the presentation linked above.

Improve this question
$\endgroup$
7
  • $\begingroup$ Left hand side of yellow block is conditional expectation or truncated mean, it is not mean or expected value of $x$. Please re look the things may be square bracket exactly represents what I say. $\endgroup$
    – SAAN
    Commented Aug 9, 2013 at 14:24
  • 1
    $\begingroup$ that doesn't look quite right to me. How can you have $y_i$ on the right-hand side but not on the left hand side? I'd expect the $x$ on the right-hand side to get integrated out somehow leaving a function of $y_i$. And shouldn't it be a capital $X$, for a random variable, on the left hand side, also? Are you sure the left hand side isn't something like $\rm{E}(X|y_i)$? $\endgroup$
    – TooTone
    Commented Aug 9, 2013 at 14:31
  • $\begingroup$ @Azeem What evidence is there in any of these formulas or diagrams for "truncation"? $\endgroup$
    – whuber ♦
    Commented Aug 9, 2013 at 14:31
  • 2
    $\begingroup$ For reference, the slide above comes from Thomas Minka's slides here: research.microsoft.com/en-us/um/people/minka/papers/ep/… $\endgroup$
    – Ed Dean
    Commented Aug 9, 2013 at 14:37
  • $\begingroup$ @whuber: I used "or" because conditional moments and truncated moments possess same definition. $\endgroup$
    – SAAN
    Commented Aug 9, 2013 at 14:40

2 Answers 2

Reset to default
5
$\begingroup$

$$\text{E}[X] = \frac{\int x \, P(y_i \mid x) \, dx}{\int P(y_i \mid x) \, dx}$$

is not a general statement, but only a first step in expectation propagation (EP). EP tries to approximate a posterior distribution $P(x \mid \mathcal{D})$ using a given factorization of the joint,

$$P(x) \prod_i P(y_i \mid x).$$

To reduce clutter, the dependency on the data $\mathcal{D} = \{ y_1, ..., y_n \}$ is often dropped in the notation. Instead of a posterior distribution, it might actually be less confusing to just think about approximating any distribution whose unnormalized density is given by

$$\phi_0(x) \prod_i \phi_i(x).$$

The first moment of the true distribution would be

$$\text{E}[X] = \int x \, P(x) \, dx = \frac{\int x \, \phi_0(x) \prod_i \phi_i(x) \, dx}{\int \phi_0(x) \prod_i \phi_i(x) \, dx}.$$

EP works by iteratively refining the distribution with one of the factors and approximating the distribution by only keeping some of the moments.

Improve this answer
$\endgroup$
2
$\begingroup$

The formula on that slide was a straw man and not intended to make sense. The point was that moment matching does not make sense on an individual likelihood term in isolation. This is illustrated further on the next slides. I have actually seen this bad approach used in papers, so I thought it was worth pointing out. This is one of those cases where "you had to be there."

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.