5
$\begingroup$

I am currently trying to learn the two related concepts of the Rao-Blackwell theorem and the Lehmann-Scheffé theorem.

Assume we have the random sample $X_1, \dots, X_n$ with mean $\mu$ and variance $\sigma^2 < \infty$. We have that $E[S^2] = \sigma^2$, where $S^2 = \sum_{i = 1}^n \dfrac{(X_i - \bar{X})^2}{n - 1}$ and $\bar{X} = \sum_{i = 1}^n \dfrac{X_i}{n}$. Now assume the $X_i$ are Poisson random variables with parameter $\lambda$. My understanding is that, using Lehmann-Scheffé, we get that $E[S^2 \mid \bar{X}] = \bar{X}$. Then, using the law of total variance, we get that $\text{Var}(S^2) > \text{Var}(\bar{X})$. Based on what I've read, the two above theorems imply that, if we define, say, a sufficient statistic $T_1(\mathbf{X})$ and a complete sufficient statistic $T_2(\mathbf{X})$ for some parameter $\varphi$, then, under some conditions, we can say that $\text{Var}(T_1(\mathbf{X})) > \text{Var}(T_2(\mathbf{X}))$. However, I'm having trouble understanding this last part. I've read over various notes on the subject, but I'm still not sure I understand what it's saying. Why can we say that $\text{Var}(T_1(\mathbf{X})) > \text{Var}(T_2(\mathbf{X}))$? And what are these 'conditions' that make this inequality valid?

$\endgroup$
15
  • 1
    $\begingroup$ $E[S^2|\bar{X}] = \bar{X}$??? $\endgroup$
    – AdamO
    Apr 14 '21 at 17:10
  • 2
    $\begingroup$ I can't think of an example of an incomplete and sufficient statistic and another complete and sufficient statistic for the same probability model. R-B says if you take an unbiased statistic and condition on the sufficient statistic, you will get a new estimator with lower variance but there may be other unbiased estimators with even lower variance. L-S says if it's complete and sufficient, you get the UMVUE. $\endgroup$
    – AdamO
    Apr 14 '21 at 17:15
  • 1
    $\begingroup$ I think the Hodge's Superefficient estimator is a simple counter example that actually helps understand how ill-formed some of this early variance bound research was: en.wikipedia.org/wiki/Hodges%27_estimator $\endgroup$
    – AdamO
    Apr 14 '21 at 17:21
  • 1
    $\begingroup$ @AdamO Sorry about that. I've been trying to read from multiple documents explaining the same thing, so I got confused when typing out my question. Is that better? $\endgroup$ Apr 14 '21 at 17:31
  • 1
    $\begingroup$ @SextusEmpiricus Oh my goodness, you're right. It seems that I completely screwed it up. The problem is that this was all based on memory, so I was trying to recall the idea and then ask. And it seems that, in doing so, I confused myself and wrote the wrong thing! My apologies! Given what I have written, it seems that what Thomas wrote is absolutely correct. Again, I apologise for any confusion. $\endgroup$ Apr 22 '21 at 5:58
6
+100
$\begingroup$

There is a complete sufficient statistic for $\theta$ in a model ${\cal P}_\theta$ if and only if the minimal sufficient statistic is complete (according to Lehmann "An Interpretation of Completeness and Basu’s Theorem"). This means you can't have distinct $T_1(X)$ and $T_2(X)$ the way you want. As the paper says (first complete paragraph of the second column)

On the other hand, existence of a complete sufficient statistic is equivalent to the completeness of the minimal sufficient statistic, and hence is a property of the model ${\cal P}$.

That is, in any given model, if $T_2$ is complete sufficient and $T_1$ is sufficient, $T_1$ is also complete sufficient.

The two theorems say

1/ Rao-Blackwell: Conditioning on any sufficient statistic will reduce the variance. This follows from the law of total variance

2/ Lehmann-Scheffé: In the special case that the model has a complete sufficient statistic, you get a fully efficient estimator.

In the Poisson case, the minimal sufficient statistic $\bar X$ is complete and the two estimators are identical.

There's an interesting example here of a situation where a Rao-Blackwell-type estimator is not fully efficient (not even admissible). The model is $X\sim U[\theta(1-k),\theta(1+k)]$ for known $k$ and unknown $\theta$. The Cramér-Rao bound does not apply, since the range of $X$ depends on $\theta$. A sufficient statistic is the pair $(\min X_1, \max X_n)$, and any observation is an unbiased estimator, however $E[X_1|(\min X_1, \max X_n)]$ is not even the best linear function of the two components of the sufficient statistic.

A couple more points to fill in potential gaps:

  1. This leaves open is whether there might be an unbiased estimator attaining the Cramér-Rao bound that isn't obtainable using the Lehmann-Scheffé theorem. There isn't (in reasonably nice models): any model where the bound is attained has a score function of the form $$\frac{\partial \ell}{\partial \theta}=I(\theta)(g(x)-\theta)$$ for some $g()$ (where $I()$ is the information), in which case $g(x)$ is a both a complete sufficient statistic for $\theta$ and the minimum variance unbiased estimator.

  2. As @AdamO indicates, none of this translates tidily to asymptotics: there are asymptotically unbiased estimators that beat the asymptotic information bound at a point (Hodges superefficient estimator) and even on a dense set of measure zero(Le Cam's extension of Hodge's estimator). The best you can do is the Local Asymptotic Minimax theorem, which says you can't beat an 'efficient' estimator uniformly over neighbourhoods of $\theta_0$ with diameter $O(n^{-1/2})$

$\endgroup$
5
  • $\begingroup$ Thanks again, Thomas. Are you sure about what you said in the first part? Isn't $E[S^2 \mid \bar{X}] = \bar{X}$ the special case of Lehmann-Scheffé? I was told that, if we have two statistics, $T_2(\mathbf{X})$ and $T_1(\mathbf{X})$ are a complete sufficient statistic and statistic, respectively, for some parameter of interest $\theta$, then $\text{Var}(T_1(\mathbf{X})) > \text{Var}(T_2(\mathbf{X}))$. So my question is then as follows: under what conditions does this inequality hold? $\endgroup$ Apr 19 '21 at 18:49
  • $\begingroup$ The point is it never holds for the same $\theta$. If $T_2$ is complete sufficient and $T_1$ is sufficient, then $T_1$ will also be complete. $\endgroup$ Apr 19 '21 at 21:51
  • $\begingroup$ Hmm, that's very interesting. Is there are theorem that says this, or something that I can point to to justify this reasoning ("this never holds for the same $\theta$, because ...")? $\endgroup$ Apr 19 '21 at 21:53
  • 2
    $\begingroup$ The first link is to a paper by Lehmann that says the minimal sufficient statistic is complete in any model with a complete sufficient statistic $\endgroup$ Apr 20 '21 at 1:15
  • $\begingroup$ This is a great answer, thanks! I was very confused as to how to know when a Rao-Blackwell-type estimator is an MVUE; I see now that completeness is the key. Are there any results on when statistics obtained from factorization are complete? $\endgroup$ Jun 21 '21 at 21:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.