# Why does the Rao-Blackwell Theorem require $\Bbb E(\hat{\theta}^2) < \infty$?

The Rao-Blackwell Theorem states

Let $\hat{\theta}$ be an estimator of $\theta$ with $\Bbb E (\hat{\theta}^2) < \infty$ for all $\theta$. Suppose that $T$ is sufficient for $\theta$, and let $\theta ^ * = \Bbb E (\hat{\theta}|T)$ Then for all $\theta$, $$\Bbb E (\theta^* - \theta )^2 \leq \Bbb E (\hat{\theta} - \theta )^2$$ The inequality is strict unless $\hat{\theta}$ is a function of $T$

If I understand this theorem correctly, this states that, if I have a sufficient statistic $T$ for $\theta$, then the conditional expected value of $\hat{\theta}$ given $T$ is the solution to $\min_{\hat{\theta}} \Bbb E $$(\hat{\theta}-\theta)^2 My Quesitons 1. Am I correct that \theta^* minimizes \Bbb E$$(\hat{\theta}-\theta)^2$ ?
2. Why does the Rao-Blackwell Theorem require $\Bbb E(\hat{\theta}^2) < \infty$?
3. Why is the inequality strict unless $\hat{\theta}$ is a function of $T$ ?

## 2 Answers

1. No, $\theta^*$ is a better estimator than $\hat\theta$ but not necessarily the best (whatever that means!)
2. If the estimator has no variance, then its risk is infinite and there is no guarantee that $\theta^*$ has a finite risk (even though this may happen as pointed out by Horst Grünbusch in his comments).
3. Under finite variance for $\hat\theta$, the inequality is strict because of the variance decomposition as the sum of the expected conditional variance plus the variance of the conditional expectation $$\text{var}(\hat\theta)=\mathbb{E}_T[\text{var}(\hat\theta|T)]+ \text{var}_T(\mathbb{E}[\hat\theta|T])=\mathbb{E}_T[\text{var}(\theta|T)]+\text{var}_T(\theta^*)$$ Unless the expected conditional variance is zero, which amounts to $\hat\theta$ a function of $T$ only.
• ad 2: Why is it impossible that $E(\hat{\theta}^2|T)<E(\hat{\theta}^2)=\infty$? Consider $\hat{\theta} = X + C$ as estimator for $\mu$, where $X \sim \mathcal{N}(\mu,\sigma^2)$ , and $C$ an unrelated Cauchy-distributed rv. – Horst Grünbusch Feb 26 '16 at 14:17
• @HorstGrünbusch Why would the Cauchy piece go away when you condition on $T$? Also $\hat{\theta}$ is not an unbiased estimator. – dsaxton Feb 26 '16 at 15:51
• @HorstGrünbusch It seems to me that your $\hat{\theta}$ does not even have a conditional expectation $\mid T$ (since $C$ does not have an expectation), thus $\theta^\ast$ would be undefined. – Juho Kokkala Feb 26 '16 at 15:53
• OK, all I wanted was $C$ without variance, not without expectation. ;) Now take $C \sim t_2$, i.e. Student-t-distributed with 2 degrees of freedom and $E(C)=0$ and $C$ independent of $X$. Sufficient statistic is clearly $X$. Then $E(X+C|X) = E(X|X) + E(C|X) = X + E(C) = X$, but $\infty=Var(C) + Var(X) = Var(X+C)>Var(X+C|X)=\sigma^2$ – Horst Grünbusch Feb 26 '16 at 16:23
• So I think it's wrong that a Rao-Blackwell estimator has necessarily infinite variance if the original estimator has infinite variance. (Yet even if both variances would necessarily be infinite $\infty \leq \infty$ would still hold.) – Horst Grünbusch Feb 26 '16 at 16:36
1. Note that being a sufficient statistic is not unique. Trivially, the whole data are sufficient, but conditioning an estimator on them doesn't change anything. So a sufficient statistic alone is not sufficient (pun!) for having minimal mean squared error. See the Lehmann-Scheffé-theorem, which uses the Rao-Blackwell-theorem in the proof, for a sufficient sufficiency (in fact, being sufficient and complete).

2. If both are infinite, the weak inequality is always true. But then, as a counterexample, you can construct a sufficient statistic that is not a function of $T$ but has still infinite variance (such that only $\leq$ holds).

Take for example $C_1 \sim t_2 + \mu$, a shifted $t_2$-distributed random variable with $E(C_1) = \mu$ and $Var(C_1) = \infty$, and as another independent random variable $C_2 \sim t_2$. The parameter to estimate is $\mu$. Original estimator is $\hat{\theta} = C_1 + C_2$. A sufficient statistic is of course $C_1$. Both the Rao-Blackwell estimator $E(\hat{\theta}|C_1)=C_1$ and $\hat{\theta}$ have infinite variance. So the inequality would hold weakly. On the other hand, $C_1+C_2$ is not a mere function of $C_1$: It involves the other random variable, so that would be a contradiction to the last sentence you asked your 3rd question about. In fact, some textbooks admit infinite variance for the original estimator, but in turn they cannot state when $<$ holds.

1. If $\hat{\theta}$ is a function of $T$, you can prove by the factorization theorem that $\hat{\theta}$ is already sufficient for $\theta$. So again we end up with improving nothing. Apart from this case, the inequality is strict, and that's the non-trivial assertion of the theorem.