The Rao-Blackwell Theorem states

Let $\hat{\theta}$ be an estimator of $\theta$ with $\Bbb E (\hat{\theta}^2) < \infty$ for all $\theta$. Suppose that $T$ is sufficient for $\theta$, and let $\theta ^ * = \Bbb E (\hat{\theta}|T)$ Then for all $\theta$, $$\Bbb E (\theta^* - \theta )^2 \leq \Bbb E (\hat{\theta} - \theta )^2$$ The inequality is strict unless $\hat{\theta}$ is a function of $T$

If I understand this theorem correctly, this states that, if I have a sufficient statistic $T$ for $\theta$, then the conditional expected value of $\hat{\theta}$ given $T$ is the solution to $\min_{\hat{\theta}} \Bbb E $$(\hat{\theta}-\theta)^2$

My Quesitons

  1. Am I correct that $\theta^*$ minimizes $\Bbb E $$(\hat{\theta}-\theta)^2$ ?
  2. Why does the Rao-Blackwell Theorem require $\Bbb E(\hat{\theta}^2) < \infty$?
  3. Why is the inequality strict unless $\hat{\theta}$ is a function of $T$ ?

2 Answers 2

  1. No, $\theta^*$ is a better estimator than $\hat\theta$ but not necessarily the best (whatever that means!)
  2. If the estimator has no variance, then its risk is infinite and there is no guarantee that $\theta^*$ has a finite risk (even though this may happen as pointed out by Horst Grünbusch in his comments).
  3. Under finite variance for $\hat\theta$, the inequality is strict because of the variance decomposition as the sum of the expected conditional variance plus the variance of the conditional expectation $$\text{var}(\hat\theta)=\mathbb{E}_T[\text{var}(\hat\theta|T)]+ \text{var}_T(\mathbb{E}[\hat\theta|T])=\mathbb{E}_T[\text{var}(\theta|T)]+\text{var}_T(\theta^*)$$ Unless the expected conditional variance is zero, which amounts to $\hat\theta$ a function of $T$ only.
  • 1
    $\begingroup$ ad 2: Why is it impossible that $E(\hat{\theta}^2|T)<E(\hat{\theta}^2)=\infty$? Consider $\hat{\theta} = X + C$ as estimator for $\mu$, where $X \sim \mathcal{N}(\mu,\sigma^2)$ , and $C$ an unrelated Cauchy-distributed rv. $\endgroup$ Commented Feb 26, 2016 at 14:17
  • 1
    $\begingroup$ @HorstGrünbusch Why would the Cauchy piece go away when you condition on $T$? Also $\hat{\theta}$ is not an unbiased estimator. $\endgroup$
    – dsaxton
    Commented Feb 26, 2016 at 15:51
  • 1
    $\begingroup$ @HorstGrünbusch It seems to me that your $\hat{\theta}$ does not even have a conditional expectation $\mid T$ (since $C$ does not have an expectation), thus $\theta^\ast$ would be undefined. $\endgroup$ Commented Feb 26, 2016 at 15:53
  • 2
    $\begingroup$ OK, all I wanted was $C$ without variance, not without expectation. ;) Now take $C \sim t_2$, i.e. Student-t-distributed with 2 degrees of freedom and $E(C)=0$ and $C$ independent of $X$. Sufficient statistic is clearly $X$. Then $E(X+C|X) = E(X|X) + E(C|X) = X + E(C) = X$, but $\infty=Var(C) + Var(X) = Var(X+C)>Var(X+C|X)=\sigma^2$ $\endgroup$ Commented Feb 26, 2016 at 16:23
  • $\begingroup$ So I think it's wrong that a Rao-Blackwell estimator has necessarily infinite variance if the original estimator has infinite variance. (Yet even if both variances would necessarily be infinite $\infty \leq \infty$ would still hold.) $\endgroup$ Commented Feb 26, 2016 at 16:36
  1. Note that being a sufficient statistic is not unique. Trivially, the whole data are sufficient, but conditioning an estimator on them doesn't change anything. So a sufficient statistic alone is not sufficient (pun!) for having minimal mean squared error. See the Lehmann-Scheffé-theorem, which uses the Rao-Blackwell-theorem in the proof, for a sufficient sufficiency (in fact, being sufficient and complete).

  2. If both are infinite, the weak inequality is always true. But then, as a counterexample, you can construct a sufficient statistic that is not a function of $T$ but has still infinite variance (such that only $\leq$ holds).

Take for example $C_1 \sim t_2 + \mu$, a shifted $t_2$-distributed random variable with $E(C_1) = \mu$ and $Var(C_1) = \infty$, and as another independent random variable $C_2 \sim t_2$. The parameter to estimate is $\mu$. Original estimator is $\hat{\theta} = C_1 + C_2$. A sufficient statistic is of course $C_1$. Both the Rao-Blackwell estimator $E(\hat{\theta}|C_1)=C_1$ and $\hat{\theta}$ have infinite variance. So the inequality would hold weakly. On the other hand, $C_1+C_2$ is not a mere function of $C_1$: It involves the other random variable, so that would be a contradiction to the last sentence you asked your 3rd question about. In fact, some textbooks admit infinite variance for the original estimator, but in turn they cannot state when $<$ holds.

  1. If $\hat{\theta}$ is a function of $T$, you can prove by the factorization theorem that $\hat{\theta}$ is already sufficient for $\theta$. So again we end up with improving nothing. Apart from this case, the inequality is strict, and that's the non-trivial assertion of the theorem.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.