I'm trying to follow ML lesson about Bayesian inference. They have n samples from a uniform distribution $U(0, \theta)$, and they suggest 2 estimators:

  1. 2 times the average $2\frac{\sum{x_i}}{n}$
  2. largest sample $x_{(n)}$


They claim that the squared errors of each estimator is:

  1. $E[(2\frac{\sum{x_i}}{n} - \theta)^2] = ... = \frac{\theta^2}{12n}$
  2. $E[(x_{(n)} - \theta)^2] = ... = \frac{2\theta^2}{(n+1)(n+2)}$

I can follow the derivation (denoted by ...), but I'm not sure what's the meaning of the left hand sides of those 2 formulas? why isn't it $E[(x_{i} - \theta)^2]$ always? it looks as if they subtract each estimator from itself :S

  • 1
    $\begingroup$ You really ought to explain what "$\theta$" stands for. But going to your question: if you were to compute $\mathbb{E}[(x_i-\theta)^2]$, what would that have to do with any of the estimators?? $\endgroup$ – whuber Nov 4 '16 at 17:38
  • $\begingroup$ Agree with @whuber, and I'm going to guess $\theta$ is the maximum since you're considering $x_{(n)}$ an estimator. Why don't you think in terms of the bias-variance decomposition for squared error? That is, $\text{MSE}(\hat{\theta}) = \text{Var}(\hat{\theta}) + \text{Bias}(\hat{\theta})^2$. $\endgroup$ – dsaxton Nov 4 '16 at 17:56

Suppose iid observations in your sample $\boldsymbol x$ are drawn from a uniform distribution; i.e. $$X_i \sim \operatorname{Uniform}(0,\theta), \quad \theta > 0,$$ where $\theta$ is an unknown parameter. We are interested in the behavior of various estimators $w(\boldsymbol x)$ of $\theta$, specifically with respect to their mean squared error, defined by $$\operatorname{MSE}[w] = \operatorname{E}[(w(\boldsymbol x) - \theta)^2].$$ A natural choice of point estimator of $\theta$ follows from the minimal sufficient statistic $T(\boldsymbol x) = x_{(n)}$, the largest observation in the sample; this also happens to be the maximum likelihood estimator $\hat \theta_{\text{MLE}}$. An alternative estimator is twice the sample mean, which is the method of moments estimator $$\tilde \theta_{\text{MOM}} = 2\bar x,$$ a fact that is shown by equating the sample mean with the population mean. The goal of the exercise is to derive the MSE of each estimator and compare them. (It may also be worthwhile to compare their biases but this is not in the scope of your question.)

Your immediate question pertains to a misunderstanding of the notation. The parameter $\theta$ is fixed and unknown and not a random variable. The sample is a random variable, and any statistic derived from that sample (two of which we have discussed) is also random. An estimator $w(\boldsymbol x)$ is a statistic; it is a random variable that is calculated from the sample. Under the distributional assumption stated at the very beginning of this answer (i.e. that observations are iid and drawn from a uniform distribution with unknown maximum), we are able to compute moments of various functions of the sample; that is to say, the moments of the chosen estimator or functions of it. The result will be a function of the unknown parameter and possibly the sample size $n$, but will not be a function of any random variable or of the sample itself.

The meaning of the expression you wrote, $$\operatorname{E}[(x_i - \theta)^2]$$ is simply the expected squared difference from $\theta$ of a single (specifically, the $i^{\rm th}$) observation from the sample $\boldsymbol x = (x_1, \ldots, x_n)$. Noting that $\operatorname{E}[x_i] = \theta/2$, we have $$\begin{align*} \operatorname{E}[(x_i - \theta)^2] &= \operatorname{E}[(x_i - \theta/2)^2 - \theta(x_i - \theta/2) + (\theta/2)^2] \\ &= \operatorname{E}[(x_i - \theta/2)^2] - 0 + (\theta/2)^2 \\ &= \operatorname{Var}[x_i] + \frac{\theta^2}{4}. \end{align*}$$ Knowing that the variance of the uniform distribution on $[0,\theta]$ is $\theta^2/12$, we could write this as $\theta^2/3$. This would be the mean squared error of the estimator $w(\boldsymbol x) = x_i$, and it is quite clearly not a good estimator of $\theta$, since the error is not a decreasing function of the sample size, and is indeed totally independent of the sample size. This should not come as a surprise: this choice of estimator ignores all but one observation in your sample.

  • 2
    $\begingroup$ @ihadanny If you look back at what you wrote, you will see that you wrote $\theta$, not $\theta/2$. Remember, you wanted to find the MSE of an estimator for $\theta$, not of an estimator of $\theta/2$. It just so happens that for a uniform distribution, $\theta/2$ is the mean. $\endgroup$ – heropup Nov 4 '16 at 19:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.