Understanding the Rao-Blackwell Theorem

Question

I've been reading up a lot on the practical applications of the Rao-Blackwell theorem. I do understand how the Bias and Variance and MSE aspects of the theorem fall in place (i.e. the mathematical proof makes sense), but I do not understand how certain practical applications of the theorem work, based on my own university's materials and highly-recommended third-party textbooks.

These are some of the applications I came across:

Application 1

Let $X = \{X_1, X_2,\ldots,X_n\}$ be a random sample from a distribution with mean $µ$ and variance $σ^2$. Suppose that $S(X) = \sum_iX_i$ is a sufficient statistic for $µ$. We consider $\hat\mu_1 = X_1$ as an initial estimator of $µ$ and seek a better one.

The Rao–Blackwell theorem states that the following estimator is better: $$\hat\mu^b_2 = \mathbb E(\hat\mu^b_1 | S(X)) = \mathbb E[X_1 | S(X)]$$

This is equal to: \begin{align*} \mathbb E[X_1 | S(X)] &= 1/n \,\sum_i \mathbb E[X1 | S(X)]\\ &= 1/n \,\sum_i\mathbb E[(Xi | S(X)]\\ &= 1/n\, \mathbb E[\sum_iX_i | S(X)]\\ &= 1/n\, \mathbb E[\sum_iX_i | ∑_iX_i]\\ &= 1/n \sum_i X_i\\ &= \bar X_n\\ \end{align*} Note: all the $\sum_iX_i$ figures above refer to a sum over $i$ from $i=1$ to $i=n$.

So now my question is: given we had to be abstract here by not actually calculating conditional distribution probabilities, we had to solve the question above in a very, well, abstract way. First of all, how is it logically possible to equate $\mathbb E[X_1 | S(X)] = 1/n ∑_i \mathbb E[X_1 | S(X)]$? And is it okay to switch $X_1$ with $X_i$, because $X_1$ is a variable, while $X_i$ is sort of a range of variables?

Application 2

Let $X_1,\, X_2$ be i.i.d. random variables of distribution $N(θ,1)$. We have the statistic $\bar{X} = (X_1 + X_2)/2$. Now we want to try and condition our statistic on the non-sufficient statistic $X_1$ to prove we don't get a sufficient statistic in the end. So: \begin{align*} X^* &= \mathbb E[\bar X| X_1]\\ &= (1/2)\mathbb E(X_1 | X_1) + (1/2)\mathbb E(X_2 | X_1)\\ &= (1/2)(X_1) + (1/2)(θ) \end{align*} So they successfully proved $X^*$ is not sufficient. But how did $(1/2)\mathbb E(X_1 | X_1)$ equal $X_1$? Shouldn't it be $θ$?

Application 3

Let X = {X1, X2, ..., Xn} be a random sample from the Bernoulli(π) distribution. We now use the Rao–Blackwell theorem to find an estimator of π which is an improvement on X1.

So we already know ∑X is a sufficient statistic for π. Thus, we are trying to find E(X1 | ∑X). BUT FIRST, consider:

∑E(Xj | ∑Xi) = E(∑Xj | ∑Xi) = E(∑Xj | ∑Xi) = ∑Xi = nx̄

And ∑E(Xj | ∑Xi) = nE(X1 | ∑Xi)

So E(X1 | ∑Xi) = x̄

And like this, they have proved x̄ is a better estimator than just X1. But in the first line, when they switched ∑E(Xj | ∑Xi) with E(∑Xj | ∑Xi), are they talking about switching to E(∑(Xj | ∑Xi)), or E((∑Xj) | ∑Xi)? And is is true that we can substitute ∑E(Xj) = nE(X1)?

So that's about it. Th example of Rao-Blackwellization in Wikipedia and other sources seems to make sense. But if anybody has any idea of how to solve these (might-be-very-silly!) questions, then I would greatly appreciate it.

Answer 1

$\newcommand{\E}{\operatorname{E}}$In your application 1, adding more details gives $$ \E[X_1\mid S(X)] = \frac 1n \cdot n \cdot \E[X_1\mid S(X)] = \frac 1n \sum_{i=1}^n \E[X_1\mid S(X)]. $$ Now since the $X_i$ are an iid sample we have $\E[X_1\mid S(X)] = \E[X_2\mid S(X)] = \dots = \E[X_n\mid S(X)]$ (i.e. it didn't matter that we started with $X_1$. We could have started with any of the $X_i$ so effectively this is a relabeling and iid samples aren't affected by relabeling) so we replace the $i$th $\E[X_1\mid S(X)]$ with $\E[X_i\mid S(X)]$ to get $$ \E[X_1\mid S(X)] = \frac 1n \sum_{i=1}^n \E[X_i\mid S(X)]. $$ Then the linearity of expectation gives us $$ \frac 1n \sum_{i=1}^n \E[X_i\mid S(X)] = \E[\bar X_n \mid S(X)] = \bar X_n $$ since $S(X) = \bar X_n$ (more formally, $\bar X_n$ is measurable with respect to the $\sigma$-algebra generated by $S(X)$ so no "averaging" happens). This shows that the function $\E[X_1\mid S(X)]$ was actually $\bar X_n$ all along.

---

For application 2 it's the same idea in that there's no "averaging" in $\E[X_1\mid X_1]$, meaning that $\E[X_1\mid X_1]$ is just the function $X_1$.

---

For application 3, $$ \sum_{j=1}^n \E\left[X_j \mid \sum_{i=1}^n X_i\right] = \E\left[\sum_{j=1}^n X_j \mid \sum_{i=1}^n X_i\right] $$ by linearity. But $\sum_j X_j =\sum_i X_i$ so these are the same function and again there's no "averaging" so $$ \sum_{j=1}^n \E\left[X_j \mid \sum_{i=1}^n X_i\right] = n \bar X_n. $$

---

Overall it looks like the common thread here is understanding when $\E[X\mid Y] = X$ so I'd recommend reviewing the relevant properties of conditional expectation. This property is sometimes called pulling out known factors. Here's one example of a proof of it with the discrete case. It's also important to remember that $\E[X\mid Y]$ is a function, not a number, so e.g. $\E[X_1\mid X_1]$ is still a random variable.

For a bit of a sense of why $\E[X\mid X]=X$, i.e. no reduction happens, conditional expectation takes a random variable (or more generally a $\sigma$-algebra) and produces a new random variable (that is unique with probability one) that is simpler in that it has the same "local averages" as the original variable but also doesn't change too rapidly (more technically it has sufficiently simple preimages to be measurable). In $\E[X\mid X]$ we have that $X$ has the same local averages as $X$ and also has the same preimages so $\E[X\mid X]=X$. Here's a resource for a more rigorous treatment and the wikipedia article on conditional expectation has a lot of the terms and definitions.