# Variance of Rao Blackwellization for MC Estimate of Expectation

If we have a function $$J(X,Y)$$ of two random variables $$X$$ and $$Y$$ and we want to compute the expectation $$\mathbb E_{p(X,Y)}[J(X,Y)]$$.

We define $$\hat J(X)= \mathbb E_{p(Y)}[J(X,Y)\mid X]$$.

Note that: $$\mathbb E_{p(X,Y)}[J(X,Y)] = \mathbb E_{p(X)} [\hat J(X)]$$

So we can use $$\hat J(X)$$ instead of $$J(X, Y)$$ in a Monte-Carlo Estimate.

For the variance holds (variance reduction) $$var(\hat J(X)) = var(J(X,Y)) - \mathcal E[(J(X,Y) - \hat J(X))^2]$$

Here I have one questions:

• How can we prove that (variance reduction)? In the paper, no proof is given.

Here is one proof of this question (from this link):

The variance reduction follows from the the law of total variance. Suppose that $$W,Z$$ are two random variables, then it follows that $$\mathbb{V}(W)=\mathbb{V}(\mathbb{E}(W\vert Z))+\mathbb{E}(\mathbb{V}(W\vert Z))$$ then, replace $$W$$ by $$J(X,Y)$$ and $$\mathbb{E}(W\vert Z)$$ by $$\hat{J}(X)$$ and we obtain: $$\mathbb{V}(J(X,Y))=\mathbb{V}(\hat{J}(X))+\mathbb{E}(\mathbb{V}(J(X,Y)\vert X))$$ Notice that the second summand on the right hand side is given by $$\mathbb{V}(J(X,Y)\vert X)=\mathbb{E}(J(X,Y)^2\vert X)-(\mathbb{E}J(X,Y)\vert X)^2=\mathbb{E}(J(X,Y)^2\vert X)-\hat{J}(X)^2$$ plug into the ANOVA identity, solve with respect to $$\mathbb{V}(\hat{J}(X)$$ to obtain $$\mathbb{V}(\hat{J}(X))=\mathbb{V}(J(X,Y))-\left(\mathbb{E}(J(X,Y)^2)-\mathbb{E}(\hat{J}(X)^2)\right)=\mathbb{V}(J(X,Y))-\mathbb{E}\left(\left(J(X,Y)-\hat{J}(X)\right)^2\right)$$ as desired.

======================== end proof ==========================

But the can't follow the idea of the last line: $$\mathbb{V}(J(X,Y))-\left(\mathbb{E}(J(X,Y)^2)-\mathbb{E}(\hat{J}(X)^2)\right)=\mathbb{V}(J(X,Y))-\mathbb{E}\left(\left(J(X,Y)-\hat{J}(X)\right)^2\right)$$ why the following equality holds? $$\mathbb{E}(J(X,Y)^2)-\mathbb{E}(\hat{J}(X)^2) = \mathbb{E}\left(\left(J(X,Y)-\hat{J}(X)\right)^2\right)$$

Thanks.

## 2 Answers

Another form of answer can be the following, based on the proof outlined in the post you've linked:

If you open up the square form: $$\mathbb E[(J(X,Y)-\hat J(X))^2]=\mathbb E[J(X,Y)^2]-2\mathbb E[J(X,Y)\hat J (X)]+\mathbb E[\hat J(X)^2]$$

The middle term can be written as (using Law of Total Expectation):

$$\mathbb E[J(X,Y)\hat J (X)]=\mathbb E[\mathbb E[J(X,Y)\hat J (X)|X]]=\mathbb E[\hat J(X)\mathbb E[J(X,Y)|X]]=\mathbb E[\hat J(X)^2]$$

When substituted, you've the equality.

The Pythagorean decomposition of the variance $$\mathbb{V}(W)=\mathbb{V}(\mathbb{E}(W\vert Z))+\mathbb{E}(\mathbb{V}(W\vert Z))$$ shows that the original variance is a sum of two positive terms, hence $$\mathbb{V}(W)\ge\mathbb{V}(\mathbb{E}(W\vert Z))$$ showing that both versions have the same expectation and that the conditional expectation has a smaller variance.