2
$\begingroup$

Let $Y_1,\ldots, Y_n$ be independent and $N(x_i\theta,1)$ distributed, with for each $Y_i$ a mean of $x_i\theta$ for known $x_1,\ldots,x_n$. In a previous section of this exercise I found that the Cramér–Rao lower bound for estimating $\theta$ is equal to $$1\left/\left(\sum_{i=1}^n x_i^2\right) \right.$$ Now I must find an (UMVU) estimator that has a variance equal to this lower bound.

I found the unbiased estimator $$T=\frac{\sum_{i=1}^n Y_i}{\sum_{i=1}^n x_i}$$

Indeed: $$\operatorname{E}[T]=\frac{1}{\sum_{i=1}^n x_i} \cdot \sum_{i=1}^n \operatorname{E}[Y_i]=\frac{1}{\sum_{i=1}^n x_i} \cdot \sum_{i=1}^n x_i\theta = \theta$$

However, if I'm not mistaken, the variance is

$$\mathbb{Var}[T]=\frac{1}{\left(\sum_{i=1}^n x_i \right)^2} \cdot \sum_{i=1}^n\operatorname{Var}[Y_i]=\frac{n}{\left(\sum_{i=1}^n x_i\right)^2}$$

since all the $Y_i$ are independent, and all have variance $1$.

I suppose this variance does not actually equal the required lower bound. Do I have to use another estimator, or is there actually a way to rewrite the variance above so that it's clearly equal to the lower bound? Or did I just make an error in calculation...

$\endgroup$
  • $\begingroup$ Do you know the Lehmann–Scheffé theorem or the Gauss–Markov theorem? The latter would tell you that among estimators that are linear in the vector of $y$-values, the best is the least-squares estimator. Since that is unbiased, the UMVUE, if it exists, would have to be at least as good as that. Lehmann–Scheffé would tell you that if you find the conditional expected value of any unbiased estimator (such as the one you found) given the value of a complete sufficient statistic, that will be the UMVUE. In this case, since the least-squares estimator (unlike the unbiased$\,\ldots\qquad$ $\endgroup$ – Michael Hardy Jun 11 at 1:13
  • $\begingroup$ $\ldots\,$estimator that you found) actually achieves the Cramér–Rao lower bound, you don't need to do the work of finding that conditional expected value if you take the approach of looking at the least-squares estimator and seeing that it attains the lower bound. $\qquad$ $\endgroup$ – Michael Hardy Jun 11 at 1:14
3
$\begingroup$

The complete sufficient statistic here is actually $\sum_i x_i Y_i$ and not $\sum_i Y_i$. You can see this by writing out the joint distribution in the exponential family form. As it forms a full-rank exponential family, it is easy to see $\sum_i x_i Y_i$ must be complete sufficient.

Hence if a UMVUE does exist, it must be a function of $T(X) = \sum_i x_i Y_i$. Your estimator is not a function of such a $T(X)$, and so cannot be UMVUE.

Now, clearly we have

$$ET(X) = \sum_i x_i^2 \theta$$

and so an unbiaed estimator is

$$\delta(X) = \frac{\sum_i x_i Y_i}{\sum_i x_i^2}$$

Then we have

$$\operatorname{Var}(\delta(X)) = \frac{\sum_i x_i^2 }{\left(\sum_i x_i^2\right)^2}= \frac{1}{\sum_i x_i^2}$$

So that it attains the Cramer–Rao lowerbound.

In short, your mistake was not checking that $\sum_i Y_i$ is actually a complete sufficient statistic.

$\endgroup$
  • $\begingroup$ Thank you! This makes a lot of sense. I was in fact not looking specifically for an UMVU Estimator but an unbiased one that perhaps would attain the lower bound. However, as you explain, it's smarter to look specifically for the UMVUE as that will attain exactly that, the lower bound. And to find an UMVUE estimator you also check for a sufficient and complete statistic, and by doing so, finding the right estimator will be made easier. Thanks again! $\endgroup$ – Marc Jun 11 at 9:46
  • $\begingroup$ Since the CRLB is the minimum possible variance, any unbiased estimator that attains the lower bound must be UMVUE. And under squared error loss (ie. variance) it must be unique, so it will always be a function of the complete sufficient statistic $\endgroup$ – Xiaomi Jun 11 at 13:14
2
$\begingroup$

The first thing to note here is that what you have is a simple linear regression model with a standard normal error term (i.e., having known unit variance). The model can be written as:

$$Y_i = \theta x_i + \varepsilon_i \quad \quad \quad \quad \quad \varepsilon_1,\ldots,\varepsilon_n \sim \text{IID N}(0,1).$$

The most well-known unbiased estimator of the coefficient in a linear regression model is the ordinary least-squares (OLS) estimator. Since your model has a single explanatory variable, and no intercept term, this estimator is:

$$\hat{\theta} = (\mathbf{x}^\text{T} \mathbf{x})^{-1} (\mathbf{x}^\text{T} \mathbf{y}) = \frac{\mathbf{x} \cdot \mathbf{y}}{\|\mathbf{x}\|^2} = \frac{\sum_i x_i y_i}{\sum_i x_i^2}.$$

The OLS estimator also has a well-known form for its variance:

$$\mathbb{V}(\hat{\theta}) = (\mathbf{x}^\text{T} \mathbf{x})^{-1} \cdot \mathbb{V}(\varepsilon_i) = \frac{1}{\|\mathbf{x}\|^2} \times 1 = \frac{1}{\sum_i x_i^2},$$

which is equal to your specification of the Cramér–Rao lower bound.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.